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Foreword 


What will it be like to admit Artificial Companions into our society? How will they change 
our relations with each other? How important will they be in the emotional and practical lives 
of their owners—since we know that people became emotionally dependent even on simple 
devices like the Tamagotchi? How much social life might they have in contacting each other? 


The contributors to this book assume that some form of long-term computer Companions are 
now a certainty in the coming years, and that it is a good moment to consider from a set of 
wide interdisciplinary perspectives, both how we shall construct them technically as well as 
their personal and social consequences. By Companions we mean conversationalists or 
confidants —not robots— but rather computer software agents whose function will be to get 
to know their owners, who may well be elderly or lonely, and focusing not only on 
assistance via the internet (contacts, travel, doctors etc.) that many still find hard to use, but 
also on providing company and Companionship, by offering aspects of personalization. The 
human-Companion relationship could also be used to build a life narrative of the owner, 
eliciting over a long period a structure of the owner's life, perhaps in a level of detail that 
even their relatives might not recognize, or know about. You could call that autobiography 
building for everyone, for human lives whose supporting digital data will soon all be on the 
internet (Everyone’s myriad photos, writings, travel records, videos, medical records etc.) but 
whose inner content, coherence and purpose could be elicited by long-tenn conversations 
with a Companion. Many questions will arise about the possible responsibilities of such 
Companions that leam so much about an owner: to the owner or to the state, and who will 
own those built narratives after the owner’s death? What will be the inevitable relationships 
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between Companions be like, perhaps gossiping and sharing secrets about their owners 
behind their backs? 


Computer scientists are now busy designing digital Companions: That is, virtual 
conversationalists, or digital confidants. As I noted, they will not be robots, in the classic 
sense, and this is not science fiction, but just a slightly more advanced form of what we 
already have in many laboratories, so this is a good time to stand back and reflect. 
Companions will draw technically on developments in Artificial Intelligence: language and 
interface technologies, emotion representations, image recognition, personalization 
techniques, using the human-Companion relationship to provide some conversation-based 
service for an owner, and build some representation of the owner’s life data. 


What are the techniques required to enable such conversations, such that the Companion can 
plausibly be said to understand its user, and to give a feeling of emotional Companionship 
and comfort, even an appearance of concern for the user’s needs and feelings? These are old 
questions in Artificial Intelligence but which have made considerable advance in recent 
years, theoretically and in practical artifacts. Subjects like “computers and emotion” were 
once taboo, or at least used only by a few eccentrics, in the Artificial Intelligence community 
but they have now moved to centre stage. 


These are the kinds of issues addressed by the contributions in this book. The contributors, 
who are computer and social scientists—sometimes both— assume that some form of long¬ 
term computer Companions are now a certainty in the next decade, and that it is a good 
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moment to consider their personal and social consequences from a wide set of 
interdisciplinary perspectives. Questions will arise about the technical limitations of such 
artifacts, and the social changes they will bring in, and these will be tackled in what follows 
by our expert contributors. 


The contributions here vary in length and depth, and as Plato used to note, things are not 
always better because they last longer, and there is a wide range of treatments set out here. 
The origin of the essays was a seminar held at the Oxford Internet Institute (Oil) in 2007, at 
which Sherry Turkle of MIT was our visiting speaker. I owe a debt of gratitude to Bill 
Dutton, the Director of the Oil, for hosting the event and for his encouragement in the 
production of this book, and to Joanna Carr who did much at Sheffield to make the book a 
reality, and also to Octavia Wilks for preparing the index 


Yorick Wilks 
July, 2009. 
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Section One - Setting the Scene 
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In Good Company? On the threshold of robotic Companions - Sherry 
Turkle 


Abstract 
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Most contributors to this volume believe that only technical matters stand between where we 
are now and a time when robots will be our Companions and teachers. Robots need to expand 
their domains of understanding, and if those domains should be emotional, well, that will be a 
technical matter as well. So while this volume expresses designers’ enthusiasm about robots 
as technical objects, it challenges us to see robots as something more, as evocative objects. 
What are we thinking about when we are thinking about robots? We are thinking about 
aliveness and authenticity, love and spirituality. We are thinking about what it means to build 
a psychology. We are thinking about what makes people special. Or perhaps that they are not 
so special after all. 1 


If one thinks of a classic ‘ upstairs/downstairs ’ scenario, it is no longer clear where the robots 
will be lodging. For some years, robot vacuum cleaners have staked their claim as household 
helpers, but more recently, roboticists would expand the machines domain from doing chores 
to offering Companionship and care. But to make this step, one needs a new kind of robot, 
one equipped to be good company. These are sociable robots, robots programmed to have 
mental states that shift as they interact with people. To take those already on the scene, 
consider, for example, the robotic doll My Real Baby. Bounce the doll and it gets happy; 
bounce it too much and it might get ornery. Or Pleo, a small dinosaur robot advertised for its 
psychological ‘ autonomy ’’; or Paro, a baby seal robot designed as a therapeutic Companion. 
Like the others, Paro has ‘states of mind,’’ even if primitive ones. Interacting with Paro 
requires understanding its states, a demand for vigilance that, in the end, contributes to its 
appeal. Such digital creatures are the shock troops of a cultural moment when figuring out 
how a robot is feeling ’ in order to get along with it, begins to seem a natural thing. 
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When robots make eye contact, recognize faces, mirror human gestures, they push our 
Darwinian buttons, exhibiting the kinds of behavior people associate with sentience, 
intentions, and emotions. Once people see robots as creatures, people feel a desire to nurture 
them. With this feeling comes the fantasy of reciprocation: as we begin to care for robots, we 
want them to care about us. In our nascent robotics culture, nurturance turns out to be a ‘killer 
app ’. Eleven-year-old Fara reacts to a play session with Cog, a humanoid robot at MIT that 
can meet her eyes, follow her position, and imitate her movements, by saying that she could 
never get tired of the robot because “it’s not like a toy because you can’t teach a toy; it’s like 
something that’s part of you, you know, something you love, kind of like another person, like 
a baby.” 


What seems natural to the contributors to this volume—that robots are part of our relational 
futures—is now a widely shared idea, and not just among experts. Indeed, it is so widely 
shared that it is easy to forget that, until recently, it was hardly shared at all. In the 1980s, one 
common response to the computer presence could be summed up as a ‘romantic reaction.' 2 It 
accepted that people might well be a kind of computer (a major concession to models of mind 
as program) but stressed the soul and the spirit in the human machine: simulated thinking 
might be thinking but simulated feeling is never feeling, simulated love is never love. 
Computers were fine, more than fine, if they were doing instrumental jobs, or jobs that could 
be neatly compartmentalized in the ‘ thinking ’ category. But computational objects —robots 
included—should not be allowed into the realm of human relationships. 3 So, when in 1984 I 
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called the computer a “second self’ and demonstrated people’s strong personal connection to 
them, many objected by insisting on the truism: “the computer is just a tool.” 


Over the next decade, opinions shifted. 4 Computers became everyday objects and it became 
commonplace to see one’s laptop as an extension of self. “What is on your PowerBook?” 
was a good advertising slogan in the mid-1990s because it acknowledged the degree to which 
a computer desktop reflected personal as well as intellectual commitments. People still saw 
computers as tools but recognized that they were tools with a difference. They had special 
vocations: not-yet-minds, but on the boundaries of mind, they were close to being minds 
whose opinions counted. 5 


These days, in studying reactions to sociable robots, robots that do such things as look you in 
the eye, remember your name, and track your motion, I find numbers of people who consider 
such objects as potential friends, confidants, and (as they imagine technical improvements) 
even lovers. I listen for what stands behind this new attitude and I hear three things. There is 
openness to seeing computational objects as ‘other minds'-, there is willingness to consider 
what a computer and human mind have in common; and, in a different register, there is 
evidence of a certain fatigue with the difficulties of dealing with people. A female graduate 
student comes up to me after a lecture and tells me she would gladly trade in her boyfriend 
for a sophisticated humanoid robot as long as the robot could produce what she calls “caring 
behavior.” She tells me that “I need the feeling of civility in the house and I don’t want to be 
alone.” She says: “If the robot could provide a civil environment, I would be happy to help 
produce the illusion that there is somebody really with me.” What she is looking for, she 
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says, is a “no-risk relationship ” that will stave off loneliness; a responsive robot, even if it 
were just exhibiting scripted behavior, seems better to her than a demanding boyfriend. 


The distance travelled over the past twenty years has been impressive. Bruce, a thirteen-year- 
old I interviewed in 1983, thinks of computers and robots as “perfect” and therefore different 
from flawed and frail people. Robots do everything right; people do the best they know how. 
But for Bruce, it is human imperfection that makes for the ties that bind. Specifically, his own 
limitations make him feel close to his father (“I have a lot in common with my father ... we 
both have chaos”). Perfect robots could not understand this very important relationship. 
Twenty years later, Howard, fourteen, compares his father to the idea of a robot counsellor 
and the human does not fare as well. Howard thinks that the robot would be better able to 
grasp the intricacies in the day of a high school student. He comments on what the robot 
would bring to the table: “Its database would be larger than Dad’s. Dad has knowledge of 
basic things, but not enough of high school. Robots can be made to understand things like 
‘feelings.’ 


With the belief that robots can understand them, people are, of course, more likely to warm to 
their company. In the presence of sociable robots people feel attachment and loss; they want 
to reminisce and feel loved. In a year-long study of human-robot bonding, one seventy-four- 
year-old Japanese participant said of her Wandukun, a furry robot creature designed to 
resemble a koala bear: “When I looked into his large, brown eyes, I feel in love after years of 
being quite lonely ... I swore to protect and care for the little animal.” 6 In my study of robots 
in Massachusetts nursing homes, seventy-four-year-old Jonathan responds to his My Real 
Baby robot doll by wishing it were a bit smarter because he would prefer to talk to a robot 
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about his problems than to a person. “The robot wouldn’t criticize me.” Andy, also seventy- 
four, says that the My Real Baby, which responds to caretaking by exhibiting different states 
of mind, bears a resemblance to his ex-wife Rose, “something in the eyes.” He likes chatting 
with the robot about events of the day. “When I wake up in the morning and see her face [the 
robot] over there, it makes me feel so nice, like somebody is watching over me.” 


I recently had a public exchange with colleagues who wrote about the ‘I-Thou ’ dyad of 
people and robots and I could only see Martin Buber spinning in his grave. 7 The 7’ was the 
person in the relationship, but how could the robot be the ‘Thou'1 In the past, I might have 
focused on how my colleagues’ projected feelings on a robot that definitionally could not 
have them. But I had taken that position when I interpreted attitudes toward robots as a kind 
of ‘ Rorschach ’ for better understanding people’s hopes and frustrations. Now, there was a 
new earnestness, a new literal-mindedness to the consideration of a robot Companion. My 
colleagues saw the robot in the wings and were eager to welcome it onstage. 


It seemed no time at all that a book came out called ‘Love and Sex with Robots ’ and a 
reporter from ‘Scientific American ’ was interviewing me about the psychology of robot 
marriage. 8 1 found the conversation memorable. I was asked if my opposition to people 
marrying robots didn’t put me in the same camp as those who oppose the marriage of lesbians 
or gay men. I tried to explain that just because I didn’t think people could marry machines 
didn’t mean that that any mix of people with people was fair play. The reporter accused me of 
species chauvinism. Wasn’t this the kind of talk that homophobes once used, not considering 
gays as ‘real ’ people? Our culture had clearly come to a new place. 
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To me, robots represent the new uncanny in our culture of simulation. Here I refer to the 
uncanny in Freud’s sense—something known of old and long familiar, yet now made 
strangely unfamiliar. 9 They are not like the dolls of the past that offered possibilities for pure 
projection. The new relational robots are built with psychologies and needs of their own. As 
uncanny objects, objects on the boundaries of categories, robotic creatures provoke us to ask 
questions about traditional categories, questions such as, ‘ What kinds of relationships are 
appropriate to have with machines? And more generally, ‘What is a relationship ?’ The 
question is not whether children will grow up to love their robots more than other toys, or 
indeed, their parents, but what will loving come to mean? 


The psychoanalyst Heinz Kohut describes how some people may temporarily strengthen their 
fragile sense of self by turning another person into a “self-object.” 10 In the role of self-object, 
the other is experienced as part of the self, thus in perfect tune with the fragile individual’s 
inner state. Disappointments inevitably follow. The relationships people form on social 
networking sites (whether in MySpace or Facebook chat or in virtual worlds) are excellent 
contenders for the role of self-object. There, we come to know people through a curious half- 
light, where people can be imagined to be what the fragile self needs them to be. 11 


Similarly, robots that look into your eyes, trace your movements, perhaps say your name, 
clearly present themselves as candidates for the role of self-object. If they can give the 
appearance of aliveness and yet not disappoint, they may even have a ‘ comparative 
advantage ’ over people for this job, and thus open new possibilities for narcissistic 
experience with machines. Why do I say, “comparative advantage?” When people turn other 
people into self-objects, one might say they are making an effort to turn a person into a kind 
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of ‘spare part ’ rather than taking him or her as an autonomous, individual personality. The 
artificial Companion or robot is, of course, already a spare part. 


The seductions of the robotic provide a window onto how much people are tempted to 
sidestep encounters with friends and family. Over-stressed, over-worked, people claim 
exhaustion and overload. Loneliness is failed solitude. Are cyber-connections paving the way 
to considering robotic Companions as sufficient unto the day? These days, people readily 
admit that they would rather leave a voice mail or send an email than talk face-to-face. And 
from there, they say: “I’d rather talk to the robot. Friends can be exhausting. The robot will 
always be there for me. And whenever I’m done, I can walk away.” Or as one woman said 
about AIBO, Sony’s household entertainment robot, “It is better than a real dog ... It won’t 
do dangerous things, and it won’t betray you ... Also, it won’t die suddenly and make you feel 
very sad.” The romantic reaction to the computer presence stressed that simulation had no 
place in matters of love. These days, people are likely to speculate on the possibility that 
humans, like robots, get by on simulation. One thirty-year-old puts it this way: “How do I 
know that my lover is not just simulating everything he says he feels?” A fifty-nine-year-old 
man said: “My first wife faked her orgasms for twenty-five years ... so what would make a 
robot inauthentic? She set the bar pretty low.” Or recall the graduate student who was willing 
to “help produce the illusion that there is somebody really with me” if a robot would just 
provide some background. 


In the 1980s, people insisted that the bedrock of human uniqueness was what computers 
could not do or be, placing special value on the importance of the human life cycle in 
defining what was essential about being a person. One man, considering the possibility of 
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confiding in a computer psychotherapist put it this way: “How can I talk about sibling rivalry 
to something that never had a mother?” It put a premium on the idea that only people can 
give each other understanding and empathy. It was invested in the idea that there is 
something essential about the human spirit, and that this essential quality resides in human 
inner states. Now this essentialist assumption is challenged. Today, one does not linger over 
inner states. The new focus is on behavior. What matters is how the robots perform and how 
we perfonn for each other— the essence, after all, of life in virtual communities where we 
create an avatar and put it on a self-built stage. With the focus on behavior rather than inner 
states, a creature that behaves appropriately is an appropriate creature. 


In the 1980s, debates in Artificial Intelligence centred on the question of whether machines 
could "really" be intelligent. These debates were about the objects themselves, what they 
could and could not do. Our new debates about relational and sociable machines—debates 
that will have an increasingly high profile in mainstream culture—are not only about the 
machines’ capabilities but about our vulnerabilities, both to machines that push our 
Darwinian buttons and to the promise of relationship, any relationship, in a world where 
humans so often seem to disappoint. For many who are lonely yet fearful of intimacy, a 
robotic Companion offers the illusion of Companionship without the demands of sustained, 
intimate friendship. One can be a loner yet never alone. It is a small step from this position to 
the question, one implicit in so many of the contributions in this volume: ‘ What are the 
purposes of living things? ’ 


From the perspective of today’s young people, the answers may not be obvious. Recall 
fourteen-year-old Howard who thinks that robots might be better than people when it comes 
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to understanding the intricacies of high school and eleven-year-old Fara who looks forward to 
nurturing a baby-like robot. As for me, the question has been raised very close to home. I 
took my daughter, then fourteen, to visit the Darwin exhibit at the American Museum of 
Natural History. The exhibit documented Darwin’s life and thought, and with a somewhat 
defensive tone (in light of current challenges to evolution by proponents of intelligent 
design), presented the theory of evolution as the central truth that underpins contemporary 
biology. The Darwin exhibit wanted to convince and it wanted to please. At its entrance was 
a turtle from the Galapagos Islands, a seminal object in the development of evolutionary 
theory. The turtle rested in its cage, utterly still. “They could have used a robot,” commented 
my daughter. She considered it a shame to bring the turtle all this way and put it in a cage for 
a perfonnance that draws so little on the turtle’s “aliveness.” I was startled by her comments, 
both solicitous of the imprisoned turtle because it is alive and unconcerned about its 
authenticity. The museum had been advertising these turtles as wonders, curiosities, and 
marvels—among the plastic models of life at the museum, here is the life that Darwin saw. 


I began to talk with others at the exhibit, parents and children. It was Thanksgiving weekend. 
The line was long, the crowd frozen in place. My question, “Do you care that the turtle is 
alive?” was welcome diversion. A ten-year-old girl would prefer a robot turtle because 
aliveness comes with aesthetic inconvenience: “its water looks dirty. Gross.” More usually, 
votes for the robots echoed my daughter’s sentiment that in this setting, aliveness doesn’t 
seem worth the trouble. A twelve-year-old girl opined: “For what the turtles do, you didn’t 
have to have the live ones.” Her father looked at her, uncomprehending: “But the point is that 
they are real, that’s the whole point.” 
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The Darwin exhibit gave authenticity major play: on display were the actual magnifying glass 
that Darwin used, the actual notebooks in which he recorded his observations, indeed, the 
very notebook in which he wrote the famous sentences that first described his theory of 
evolution. But, in the children’s reactions to the inert but alive Galapagos turtle, the idea of 
the ‘original’ was in crisis. I recall my daughter’s reaction when she was seven to a boat ride 
in the postcard blue Mediterranean. Already an expert in the world of simulated fish tanks, 
she saw a creature in the water, pointed to it excitedly and said: “Look mommy, a jellyfish! It 
looks so realistic!” When I told this story to a friend who was a research scientist at the Walt 
Disney Company, he was not surprised. When Animal Kingdom opened in Orlando, 
populated by ‘real, ’ that is, biological animals, its first visitors complained that these animals 
were not as “realistic” as the animatronic creatures in Disneyworld, just across the road. The 
robotic crocodiles slapped their tails, rolled their eyes, in sum, displayed ‘essence of 
crocodile ’ behavior. The biological crocodiles, like the Galapagos turtle, pretty much kept to 
themselves. What is the gold standard here? 


I have long believed that in our culture of simulation, the notion of authenticity is for us 
what sex was to the Victorians—threat and obsession, taboo and fascination. I have lived 
with this idea for many years, yet at the museum, I found the children’s position strangely 
unsettling. For them, in this context, aliveness seemed to have no intrinsic value. Rather, it 
was useful only if needed for a specific purpose. “If you put in a robot instead of the live 
turtle, do you think people should be told that the turtle is not alive?” I ask. “Not really”, said 
several of the children. Data on ‘ aliveness ’ can be shared on a ‘need to know ’ basis, for a 
purpose. This volume with its enthusiastic embrace of machines as kin both provokes and 
challenges: What indeed are the purposes of living things? 
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Introducing Artificial Companions - Yorick Wilks 

Abstract 

This introductory chapter, like the whole book itself, concerns a range of closely related 
topics: the possibility of machines having identifiable personalities, the possible future legal 
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responsibilities of such companionable machines, and the design characteristics of such 
machines, including their technical implementation and the kinds of computational theory 
they will need to embody. As will become clear, I wish to explore these topics in tenns of 
software entities, rather than robots, and in particular the sort of software agents now being 
encountered on the web, ranging at present from technical advisers to mere chatbots. I shall 
call them Companions. This introduction explores the following related aspects of a 
Companion in more detail: what kind and level of personality should be in a machine agent 
so as to be acceptable to a human user, more particularly to one who may fear technology and 
have no experience of it; and what levels of responsibility and legal attribution for 
responsibility can we expect from entities like complex web agents in the near future? 


What will an artificial Companion be like? Who will need them and how much good or harm 
will they do? Will they change our lives and social habits in the radical way technologies 
have in the past: just think of trains, phones and television? Will they force changes in the 
law so that things that are not people will be liable for damages; up till now, it has been the 
case that if a machine goes wrong, it is always the maker or the programmer, or their 
company that made it which is at fault. Above all, how many people with no knowledge of 
technology at all, such as the old or very young, will want to go about, or sit at home, with a 
Companion that may look like a furry handbag on the sofa, or a rucksack on the back, but 
which will keep track of their lives by conversation, and be their interface to the rather 
elusive mysteries we now think of as the internet or web. 


One thing we can be quite sure of is that artificial Companions are coming. In a small way 
they have already arrived and millions of people have already met them. The Japanese toys 
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known as Tamagotchi (literally Tittle eggs’) were a brief craze in the West that saw sensible 
people worrying that they had not played with their Tamagotchi for a few hours and that it 
might have begun to pine, where pining meant sad eyes and icons on a tiny screen, and 
playing with it meant no more than pushing a feed button several times. The extraordinary 
thing about the Tamagotchi (and later the US Furby) phenomenon was that people who knew 
better began to feel guilt about their behavior towards a small cheap and simple toy that could 
not even speak. 


The brief history of those toys says a great deal about people and their ability to create and 
transfer their affections, despite their possibly limited knowledge of what is really going on 
inside an object. This phenomenon is almost certainly a sign of what is to come and of how 
easy people will find it to identify with and care for automata that can talk and appear to 
remember who they are talking to. This chapter is about what it will be like when those 
objects are freely available: since most of the basic technologies such as speech recognition 
and simple machine reasoning and memory are already in place, this will not be long. Simple 
robot home helps are already available in Japan, but we only have to look at them and their 
hard plastic exteriors, and their few tinny phrases about starting the dishwasher, to realise that 
this is almost certainly not how a Companion should be. 

The technologies needed for a Companion are very near to a real trial model; some people 
think that Artificial Intelligence (AI) is a failed project after nearly fifty years, but that is not 
true at all: it is simply everywhere. It is in the computers on the two hundred tonne planes 
that land automatically in the dark and fog and which we trust with our lives; it is in chess 
programs like IBM’s Big Blue that have beaten the world champion; and it is in the machine 
translation programs that offer to translate for you any page of an Italian or Japanese news 
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chapter on the web. And where AI is certainly present, is in the computer technologies of 
speech and language: in those machine translation programs and in the typewriters that type 
from your dictation, and in the programs on the phone that recognise where you want to buy a 
train ticket to. But this is not a chapter about computer technology any more than it is about 
robots; nor is it about Philosophy. Companions are not at all about fooling us, because they 
will not pretend to be human at all. 


Imagine the following scenario, which will become the principal one running through this 
chapter. An old person sits on a sofa, and beside them is a large furry handbag, which we 
shall call a Senior Companion; it is easy to carry about, but much of the day it just sits there 
and chats. Given the experience of Tamagochi, and the fact that old people with pets survive 
longer than those without, we will expect this to be an essential lifespan and health 
improving object to own. Nor is it hard to see why this Companion that chats in an interesting 
way would become an essential possession for the growing elderly population of the EU and 
the US, the most rapidly growing segment of the population, but one relatively well provided 
with funds. 


Other Companions are just as plausible as this, such as the Junior Companion for children, 
that would most likely take the fonn of a backpack, a small and hard to remove backpack that 
always knew where the child was, and would tell a parent. But the Senior Companion will be 
our focus, not only because of its obvious social relevance and benefit, even at a low level 
of function that could easily be built with what is now available in laboratories, but because 
of the particular fit between what a Companion is and old people’s needs. 


38 



Common sense tells us that no matter what we read in the way of official encouragement, a 
large proportion of today’s old people are effectively excluded from infonnation technology, 
the web, the internet and some mobile phones because ‘they cannot learn to cope with the 
buttons’. This can be because of their particular generation or because of losses of motor skill 
with age: there are talking books in abundance now, but many otherwise intelligent old 
people cannot manipulate a tape recorder or a mobile phone, which has too many small 
controls for them with complicated and unwanted functionalities. All this is pretty well 
known and yet there is little thought as to how our growing body of old people can have 
access to at least some of the benefits of information technology without the ability to operate 
a PC or even a cell phone. 


After all, old people’s needs are real, not just to have someone to talk to, but to deal with 
correspondence from public bodies, such as city councils and utility companies demanding 
payment; the need to set up times to be visited by nurses or relatives by phone; with how to 
be sure they have taken their pills, when keeping any kind of diary may have become 
difficult; as well as with deciding what foods to order. Even when a delivery service is 
available via the internet it is sometimes impossibly difficult in practice for them to make use 
of because they cannot hear the doorbell. In all these situations one can see how a Companion 
that could talk and understand and also gain access to the Internet, to email and a mobile 
phone could become an essential mental prosthesis for an old person, one that any 
responsible society would have to support. But there are also aspects of this which are beyond 
just getting information, such as having the news blown up on the TV screen till the print is 
big enough to be read, and dealing with affairs, like paying bills from a bank account. 
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It is reliably reported that many old people spend much of their day sorting and looking over 
photographs of themselves and their families, and events in their lives, along with places they 
have lived and visited. This will obviously increase as time goes on and everyone begins to 
have access to thousands of digitized photos and videos throughout their lives. One can see 
this as an attempt to establish the narrative of one’s life; what drives the most literate segment 
of the population to write autobiographies even when, objectively speaking, they may have 
lived lives with little to report. But think of what will be needed if a huge volume of material, 
gathered over a whole digital life is to be sorted. A related notion of ‘Memories for Life’ has 
been developed and recently been declared a major challenge for future UK computing, and 
we shall return to this matter again below. 


One can see what we are discussing, then, as democratising the art of autobiography, which 
was about Chaps, as Robert E. Benchley readers will remember, whereas the art of 
Geography was about Maps. And this will mean far more than simply providing ways in 
which people with limited manipulative skills can massage photos and videos into some kind 
of order on a big glossy screen: it will require some guiding intelligence to provide and 
amplify a narrative that imposes coherence and a time order. Lives have a natural time order, 
but this is sometimes very difficult to impose and to recover for the liver; even those with no 
noticeable problems from aging find it very hard to be sure in what order two major life 
events actually happened: ‘I know I married Lily before Susan but in during which marriage 
did my father die?’ 
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That frivolous example is to illustrate, and we shall see something of how it is actually done 
later on in the book, how an artificial agent might assist in bringing events, whether in text or 
pictures, into some single coherent order, or at least some partial orders so that one knew 
some things were before others, even if there were some events (e.g. Teddy’s and Joan’s 
marriages) that could not be ordered with certainty. This is the kind of thing today’s 
computers can be surprisingly good at, but it is a very complex and abstract notion, that of the 
time ordering of events, which can be simple (I know James was born before Ronnie) or only 
partial in some situations (I know my brother’s two children were bom after my marriage and 
before my wife died, but in what order did they come?). These examples may seem odd or 
contrived, but they do represent real problems at the border of memory and reasoning for 
many people, especially in older age. 


Another reason why this notion of ordering life narratives is so important, for a Senior 
Companion, or otherwise, is that it is also a way of separating different but similar lives from 
each other on the web, and these two notions are closely related. This is jumping ahead a 
little, but many people know the experience of searching on the web for, say, ‘George Bush’ 
in Texas and finding there are about twenty-five of them who merit some attention from 
Google. Since two of them have been US Presidents and both educated at the same two 
Universities, they cannot be distinguished from each other by name, university and job 
alone, and one must then use life events, dates of birth and so on to separate them. To put the 
whole thing simply, distinguishing closely related or confusingly named people on the web 
requires something like a coherent lifeline of the events in their lives, which is the same 
notion we have been discussing as imposing coherence on the, possibly muddled, whole life 
memories of old people. 
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We have talked of Companions as specialised computers that could talk and assist needy 
groups of people such as the old. That help will almost certainly involve helping to organise 
their lives and memories but also interacting on their behalf with the electronic world outside. 
That may be as simple as using the web to find a supermarket’s home delivery service for 
groceries. More interestingly, it may involve using the web to find out what has happened to 
their old school friends and workmates, something millions already use the web for. But, as 
we have seen, we shall also need a notion of time lines and the coherence of lives on the web 
to sort out the right friends and schoolmates from the thousands of other people with the same 
names. So, the reasoning technologies we shall need to organise the life of the Companion’s 
owner may turn out to be the very same technologies - we have not really shown this yet, just 
said it - required to locate other individuals and select them out from all the personal 
information about the world’s population that fills up the web, since the web is now not just 
for describing the famous but covers potentially everyone. Two of my friends and colleagues 
who are professors of computer science have some difficulty on the web distinguishing and 
maintaining a difference between themselves and, in one case, a famous pornography 
supplier in Dallas, and in another a reasonably well- known disc jockey in Houston. 


These problems will soon become not just quirky but nonnal problems for everyone, and 
what I shall argue below is that the kind of computer agency we shall need in a Companion, 
and possibly a Companion that deals with the web for us if we are old or maybe just lazy, is 
in fact closely related to the kind of agency we shall need to deal with the web in any case, as 
it becomes more complex. The internet may become unusable for non-experts unless we have 
human-like agents to manage its complexity for us. The web itself must develop more 
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human-like characteristics at its peripheries to survive as a usable resource and technology: 
merely locating individuals on the web when a majority of the EU and US populations have a 
web presence will become far more difficult and time consuming than it is now. If this is 
right, Companions will be needed by everyone, not simply the old, the young and the 
otherwise handicapped. It is going to be impossible to conceive of the web without some kind 
of a human face. The notion of a Companion developed so far is anything but superhuman; it 
is vital to stress this because some of the public rhetoric about what Companionable 
computers will be like has come from films such as 2001, whose computer HAL is 
superhuman in knowledge and reasoning. He was a very dangerous Companion indeed, and 
prepared to be deceptive to get what he wanted which may be not at all what we want. 
Seymour Papert at MIT always argued that it was a total misconception that AI would ever 
try to model the superhuman, and that its mission, just like Stanford AI pioneer John 
McCarthy’s emphasis on the importance of common sense reasoning, was to capture the 
shorthand of reasoning, the tricks that people actually use to cope with everyday life. Only 
then would we understand the machines we have built and trained and avoid them becoming 
too clever or too dangerous. This same issue was very much behind Asimov’s Laws of 
Robotics, which set out high level principles that no robot should ever break so as to bring 
harm to humans. 


The difficulty here is fairly obvious and is put starkly in the 2001 MovieTif a robot were 
clever enough it would find a way of justifying (to itself) an unpleasant outcome for someone 
in particular, perfectly consistently with acceptable overall principles. I say ‘clever enough’, 
but doing that has also been a distinctively human characteristic throughout history: one 
thinks of all those burned for the good of their own souls and all those sacrificed in wars so 
that others might live. In the latter case, we are probably grateful for those lost in what were 
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really medical experiments, such as the early heart transplants, even though they were never 
called that. It will not be possible to ignore these questions when presenting Companions in 
more detail, in particular the issue of where responsibility and blame may lie when a 
Companion acts as a person’s agent and something goes badly wrong. At the moment, Anglo- 
American law has no real notion of any responsible entity except a human, if we exclude 
Acts of God in insurance policies. The only possible exception here is dogs, which occupy a 
special place in the English common law and seem to have certain rights and attributions of 
character separate from their owners. If one keeps a tiger, one is totally responsible for 
whatever damage it does, because it is ferae naturae, a wild beast. Dogs, however, seem to 
occupy a strange middle ground as responsible agents, and an owner may not be responsible 
unless the dog is known to be of “bad character”. We shall return to this later and argue that 
we may have here a narrow window through which we may begin to introduce notions of 
responsible machine agency, different from that of the owners and manufacturers of 
machines at the present time. 


It is easy to see the need for something like this: suppose a Companion told your grandmother 
that it was wann outside, but that when she went out into the freezing garden after hearing 
this news she caught a chill and became very ill. In such a circumstance, one might well want 
to blame someone or something and would not be happy to be told that Companions could 
not accept blame, and that if one read the small print on the Companion box one would see 
that the company had declined all blame and had required a signature on a document to that 
effect. All this may seem fanciful, and even acceptable if one’s grandmother recovered and 
the company gave the Companion a small tweak that so it never happened again. 
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This last story makes no sense at the moment, and indeed the Companion might point out 
with reason, when the maintenance man called round, that it had read the outside temperature 
electronically, and could show that it was a moderate reading and the blame should fall on the 
building maintenance staff, if anywhere. These issues will return throughout this book, but 
what is obvious already is that Companions will have to be prepared to show exactly why 
they said the things they said and offered the advice they did. 


A Companion’s memory of what it has said and done may be important but will be used only 
rarely, one hopes; though it may be necessary for it to repeat its advice at intervals with a 
recalcitrant user (‘You still haven’t taken your pills. Come on, take them now and I’ll tell you 
a joke you haven’t heard before’). As we noted, what may become a very important feature of 
Senior Companions is putting coherence into the memories of their owners: this was 
mentioned above as a way of organising memories and sorting fragments of text and old 
photographs, but there is another aspect to this which is not so much of relevance for the user 
as for their relatives later on. 


One can reasonably suppose that years of talking with an elderly user, and helping them 
organise their memories, will mean that a Companion also has access to a life story of the 
user that is has built up from those hours of conversation. The user might not ever want to 
hear any of this played back, but what should become of this biographical account that the 
Companion has, and which could exist in several forms: For example it would be a 
document the Companion could print or a collage of things said by the user and Companion 
put together coherently from recorded pieces, or even a single autobiographical account in the 
Companion’s version of the user’s own voice? This is not at all far-fetched: there are speech 
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generation packages now available that can be trained to imitate a particular person’s voice 
very accurately with plenty of training time, and time is exactly what the Companion and user 
would have. 


I would suggest that memoirs like these, produced over a long period by the Companion, are 
exactly what the user’s relatives will want after the user is dead, something much more 
helpful and less gruesome than the tiny video clips one can now find and run at the touch of a 
button on some solar powered Italian tombstones. The need for these is now greater than it 
was, as people live farther from their parents when old and see them less. Many people have 
wished they had spent more time discussing a parent’s earlier memories before their deaths - 
how one’s parents had met and fallen in love, for example - but then, suddenly, it is too late 
to ask, or the answer is unobtainable because of shyness on the part of parent or child. 


Production of this kind of limited memoir is an important role a Companion might come to 
play in society; experience may well show that old people will reveal memories and 
anecdotes to a Companion they would perhaps not feel able to tell their close relatives. 
Indeed, there is a long tradition in Artificial Intelligence of arguing that people may 
sometimes prefer machines to people in certain roles: AI pioneer Donald Michie always 
claimed that car drivers preferred traffic lights (or ‘robots’ as they are still called in some 
parts of the world) to policemen on traffic duty. 

The possibility of a Companion constructing and storing a biography for its owner raises the 
major issue of identity: is the Companion to be distinguished from the owner whose life it 
knows so well and whose voice it will almost certainly be able to imitate? This may be just 
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another version of the old joke about pets getting like their owners, since we have chosen to 
describe the Senior Companion as a kind of pet. Other forms of identity will also be touched 
on below, in particular the identity of the Companion as an agent for the owner, and its 
similarity and distinctness from the owner, while on the other hand functioning as a web 
agent in the world of electronic infonnation. We have not yet properly introduced the notion 
of a web agent. It is, very roughly, an intelligent package of software that one can encounter 
on the internet and which will give expert advice. It is now expected that web agents will 
have to make deals and transactions of all sorts with each other and learn to trust each other, 
as they already do in a rudimentary sense in the world of banking where agents in their 
thousands clinch money transactions between financial institutions. The kind visible to a user, 
but still rudimentary, are those which will search the whole web to find the cheapest source 
of, say, a particular type of camera. 


All this is for later and elsewhere, but these two issues of identity in an artificial world will 
also draw upon other ways in which identity has become an issue for Internet users and which 
are relatively well known. The first, known to most internet users is that of chat- room users 
who pretend to be what they are not; normally this is quite innocent pretence, and little more 
than hiding behind a pseudonym or the opposite sex during conversations, or artificial 
chatbots sometimes pretending to be a different kind of person, again, usually of the 
opposite sex to the user. In that sense, the Victorian game of sex-pretending, on which 
Turing based his famous imitation game for computers has come back as a 

widely played reality. The problems only arise, and they are very real, when impressionable 
people, usually children, are lured into meetings with people who have been encountered 
under false pretences. 
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The other issue of identity, which is a standard problem for web searchers, is finding too 
many people under the same name with a Google search and trying to find the right one, or 
even how many there are, a topic we already touched on above. It is a much researched 
problem at the moment how to sort out exactly how many people there are in the pages 
retrieved by Google for the same name. One can see the pressing interest here, in that many 
scientists now rate how famous they are by how many Google Scholar hits their name gets 
compared to their competitors. But how can they be sure all those retrieved are really 
themselves? If their names are Google Unique, the number of hits is correct, but this is a rare 
occurrence. 


The George Bush example mentioned above suggests that the best way to tell must require 
something like a rule that looks at dates, on the assumption that two people with the same 
name are very unlikely to have the same date of birth. Other rules will almost certainly deal 
with aspects of people, such as their occupations; which will, however, come up against 
unusual but very real cases like John Vanbrugh, the Eighteenth century playwright, and his 
separation from the Eighteenth century architect of Blenheim Palace of the same name who, 
oddly enough, were one and the same person, hard though it is for most rules (and people) to 
accept. Normally, occupations are good separators of individual identities, which is why they 
are listed in Swedish phone directories, a country with an unusually small set of family 
names. 


There is a further kind of complication in that, even if we could sort out muddled identities of 
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these kinds, given enough information, in some cases people do not agree on how many 
objects or people there are under discussion, so that it becomes a matter of conflict or, as you 
might say, individual belief, how many people are actually being talked about. In the case of 
Vanbrugh, we can imagine a strained conversation between a person sure they were two 
similarly-named individuals and someone else who knew they were not. It is, as one could 
put it, very difficult but normally just possible for people to communicate who do not agree 
on what things there are in the world, on the ontology, to use a word in its original sense that 
is now normally used to mean something quite different. How should a Companion discuss 
relatives when it was sure that Jane was Ethel’s aunt, and its owner even said that the day 
before, but who is convinced today that they are quite different people? This is a deep matter 
to which we shall return. 


Problems of identity will arise both in the context of representing individuals in a 
Companion’s world, which is very much based on that of its user, one that the Companion 
seeks to learn, but also in the wider world of information, which for convenience we will 
often identify with what can be found on the web or Internet. It is quite normal now to hear 
people speak of ‘virtual worlds’ in connection with the web and Internet, although it is 
usually unclear exactly what they have in mind. The only obvious place where virtual worlds 
belong naturally is in alternative worlds like second life, and of computer games, whose 
interaction with the web we will also need to consider, certainly in the case of Companions 
for the young, who spend more time in games worlds than the rest of the population. 


The general interest here is in the interaction of these considerations with identity itself: 
having a verifiable identity is part of what it means to be a human being, or at least a modem 
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human being. If a Companion is to have human-like characteristics, one will want to explore 
how its identity can have human-like features, as opposed to machine-like features where 
identity is usually a trivial matter: a car is identified uniquely simply by the numbers stamped 
on its chassis and its engine, and there is no interesting issue about that outside the world of 
car fraud. 


If a Companion is to be an interface to the Internet, say, for a user who is technologically 
incompetent yet who must conform to the standards of identity that society requires and 
imposes, then the Companion will have to understand identity to some degree and possibly be 
able to manipulate slightly different forms of it. In the US and UK, identity is currently 
established by a range of items with numbers, from passports through credit cards to health, 
driving license and tax numbers (some with associated passwords or PINs), with the Social 
Security number having a definite primacy in the US. In most EU countries there is a single 
ID number, of which the clearest example is the lifelong single Personnummer in Sweden. 
States prefer a citizen to be identified by a single number, and in the UK there is currently 
strong pressure for something closer to the Swedish model, although UK law has at the 
moment no clear definition of identity (with legally registered unique names and addresses, 
as in most of the EU): there is no legal problem in the UK with having a number of identities 
simultaneously and bank accounts for each (as a famous case brought, and lost, by the Post 
Office showed some years ago) so long as there is no intention to defraud. 


All this is important since identity checks are the basis of all Internet transactions, and if a 
Companion is to deal with a person’s affairs it will need something approaching a power of 
attorney, or at least an understanding of how identity is established in web transactions, as 
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well as a method for establishing that its owner approves of what it is doing in individual 
transactions, in case of later disputes, e.g. by angry relatives after an old person’s money has 
been spent. A scenario has recently been described in which security of identity can be 
achieved without the imposition of unique identity, which security- minded authorities would 
certainly resist but which may be very important to someone who feels they have a right to 
buy something on the internet revealing only, say, their age but not who they actually are (see 
www.chyp.com/pubwebfdes/whitechapters/chyp_ response.pdf). 


All of these issues are the subjects of active research programmes, but the scene within which 
I am setting here, is the possibility that a Companion, as a new kind if artifact among us, may 
focus our minds on a set of intellectual and practical issues in a new way, even though some 
of them are very traditional issues indeed. Details of the Authors Companions project can be 
found at www.companions -project.org. 


Section Two - Ethical and Philosophical Issues 
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Artificial Companions and their philosophical challenges - Luciano Floridi 

Abstract 

The technology for Artificial Companions is already largely available, and the question is 
when rather than whether ACs will become commodities (Benyon and Mival (2007). The 
difficulties are still formidable, but they are not insurmountable. On the contrary, they seem 
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rather well-understood, and the path from theoretical problems to technical solutions looks 
steep but climbable. 12 In the following pages, I wish to concentrate not on the technological 
challenges, which are important, but on some philosophical issues that a growing population 
of AC will make increasingly pressing. 


At the beginning of Much Ado About Nothing, Beatrice asks “Who is his Companion now?” 
These days, the answer could easily be “an artificial agent”. Artificial Companions (Lee et 
al. (2007), henceforth ACs) come in various fonns. Examples include the Wi-Fi enabled 
rabbit Nabaztag, the therapeutic robot baby-harp seal Paro (Wada and Shibata (2007)), the 
child-sized humanoid robot KASPAR (Cole (2007) or the interactive doll Primo Puel. This 
first generation of simple ACs is interactively sociable, informationally skilled and capable of 
some basic natural-language processing (AISB (2005). Later generations are expected to 
become more autonomous, and hence behave in self-initiated, self-regulated, goal-oriented 
ways, and to be able to learn from their users, in the machine-learning sense of the expression 
(Wilks (2007).. 13 


Bandai, interestingly the producer of the earlier Tamagotchi, has sold more than one million 
copies of Primo Puel since 2000. ACs are a technological success not because they are the 
outcome of some unforeseeable breakthrough in Good Old-Fashioned AI, but the social 
equivalent of Deep Blue : they can deal successfully with their interactive tasks, even if they 
have the intelligence of a toaster. And they are philosophically significant precisely because 
they are neither Asimov’s robots nor HaV s children. Out of the realm of thought experiments 
and unrestrained speculations, they posit very concrete, ethical challenges (Floridi and 
Sanders (2004), Floridi (2007), which usher in what may be defined as a fourth revolution in 
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humanity’s self-understanding (Floridi (2008). Let me explain. How we build, conceptualise 
and interact with ACs will influence our future ability to address humanity’s needs and 
wishes, with a serious impact on standards of living and related economic issues. In 2007, for 
example, an estimated $40.8 billion was spent on biological pets in the U.S. alone. 14 The 
arrival of a whole population of helpful and psychologically acceptable ACs may change this 
dramatically. 


It is often argued that ACs will become increasingly popular the more they are able to assist 
elderly users satisfactorily and cost-efficiently (Mival et al. (2004). This is true and 
encouraging, especially for countries where there is an aging population, like Japan and parts 
of Europe. However, we should also remember that technology, demography and IT-skills 
follow converging lines of development. Future generations of senior citizens will not be ‘e- 
migrants’ but children of the digital era. To them, it will be natural and unproblematic to be 
in touch with artificial agents and to be related to the world through them. The more the 
threshold between online (or life-on-line) and “onlife” (or life-off-line) becomes blurred, the 
easier it will be to accept and be able to socialise with and through synthetic, hybrid, artificial 
Companions. Here the gaming industry provides useful projections. Today, “sixty-seven 
percent of American heads of households play computer and video games” and “the average 
game player is 33 years old and has been playing games for 12 years”. 15 When they retire, it 
is not so much that they will be unable to use IT products, as that they may need help to do 
so, in the same way that one may still be perfectly able to read, but no longer without glasses. 
Thus, they may welcome the support of a personal assistant in the form of an AC, which can 
act as an interface to the rest of the world. ACs should be planned more with the digitally 
impaired in mind rather than the computer illiterates. 
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The last point suggests that, in the long term, ACs may be evolving in the direction of 
specialised computer-agents, dedicated to intelligence-non-intensive (aka dumb), specific 
informational tasks, following trends already experienced in other technological industries. 
Three are already envisionable. 


First, ACs will address social needs and the human desire for emotional bonds and playful 
interactions, not unlike pets (Lee et al. (2007), thus competing with the omnipresent TV for 
attention. Here a key question is whether allowing humans to befriend ACs may be morally 
questionable. Is there something morally wrong, or mildly disturbing, or perhaps just sad in 
allowing humans to establish social relations with pet-like ACs? And why may this not be 
the case with biological pets? Could an AC be better than a child’s doll, or a senior’s 
goldfish? Is it the level and range of interactivity that matters (but then, the goldfish may not 
count) or the emotional investment that the object can invoke and justify (but then, the old 
Barbie might count)? The question casts an interesting light on human nature, and it seems to 
belong to the sort of questions asked with respect to recreational drugs. Essentially: what’s 
wrong with him? Different answers seem to be based on different philosophical 
anthropologies or conceptions of what it means to be authentically human. Should their non- 
biological nature make us discriminate against AC’s? Is it their non-biological nature that 
makes philosophers whinge? Not necessarily, if one agrees with Descartes (1996 - originally 
1641), Huxley (1893) or Wiener (1961), who argued that animals are living machines 
anyway. On the other hand, the question casts an interesting light on our understanding of 
what kind of persons we would like to be. Perhaps there is nothing wrong with pet-like ACs. 
After all, they already constitute a widespread and innocuous phenomenon. In January 2008 
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there were more than 220 million neopets online, owned by more than 150 million people. 16 
Nobody has yet raised any moral objection. 


Secondly, ACs will provide ordinary information-based services, in contexts such as 
communication, entertainment, education, training, health and safety. Like avatars, ACs are 
likely to become means to be tele-present and interact with other people at distance, as well 
as social agents in themselves. In this context, one of the challenges is that their availability 
may increase social discrimination and the digital divide (Norris (2001)). In particular, with 
respect to individuals with relevant needs or disabilities, the hope is that they will be able to 
enjoy the support of an AC, just as the Motability Scheme in the UK, for example, provides 
disabled individuals with the opportunity to own or hire powered wheelchairs and scooters at 
affordable prices. 17 Consider that ACs might easily become embedded in future technological 
artefacts engineered for mobility, as prosthetic memory agents for those with memory 
dysfunctions. As for the new generations of students, the more memories are exogenous 
rather than endogenous, the more the educational system will have to provide individuals 
with the sort of skills required to access and give sense to information. Languages (not only 
natural, but also mathematical and artificial) and culture (not mere facts, but an open-ended 
appreciation and understanding of human developments and achievements) will be crucial. 


Finally, ACs will act as “memory stewards ” (O'Hara et al. (2006)), creating and managing a 
repository of information about their owners. This is also good news. For leaving behind a 
lasting trace, has always been a popular strategy to withstand the oblivion that inevitably 
follows one’s death. Nowadays, we can all be slightly less forgettable, insofar as we succeed 
in our mnemonic DIY. This trend will grow exponentially, once ACs become commodities. 
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Storage capacity is increasing at an astonishing pace. “Between 2006 and 2010 [...] the digital 
universe will increase more than six fold from 161 exabytes to 988 exabytes”. 18 It is only a 
matter of decades before a whole life will be recordable by an AC. But then, it will not be 
long before some smart application - based on a life-time recording of someone’s voice, 
visual and auditory experiences, expressed opinions and tastes, linguistic habits, millions of 
digital documents and so forth - will be able to simulate that person, to the point where one 
may interact with her AC even after her death, without noticing, or even deliberately 
disregarding, any significant difference. A personalised AC could make one e-mortal. After 
all, an advanced, customised ELIZA (Weizenbaum (1966) could already fool many people in 
Second Life. Will there be people working in Artificial Companions centres offering services 
who can impersonate dead people? Will some future service company offer you the 
possibility of downloading enough information about you as to make a you-liza available 
even when you are not there, or even when you will no longer be there atall? Our new 
memory stewards will exacerbate old problems and pose new and difficult ones. What to 
erase, rather than what to record (as is already the case with one’s emails), the safety and 
editing of what is recorded, the availability, accessibility and transmission of the information 
recorded, its longevity, future consumption and re-playing, the management of ACs that have 
outlived their human partners, the redressing of the fine balance between the art of forgetting 
and the process of forgiving (consider post-dictatorial or post-apartheid cultures), and the 
impact that all this will have on the construction of personal and social identities, and on the 
narratives that make up people’s own past and roots. These are only some of the issues that 
will require careful handling, not only technologically, but also educationally and 
philosophically. For example, who will be the new, professional memory workers? In the 
past, the definition of a famous person could be provided in terms of someone whose 
memories were recorded and managed by professional memory workers, often artists (poets, 
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sculptors, painters, musicians, architects and so forth), chroniclers, historians or journalists. 
Nowadays, we are all famous and a little bit less mortal insofar as we succeed in being our 
own memory keepers. However, in the same way as the commodification of cameras has not 
made the profession of photographer disappear, but has reshaped its nature and our 
understanding of it, will AC’s also cause a comparable transformation? Are we going to 
witness the emergence of new professional creators and managers of digital memories? 


The previous trends suggest that ACs are part of a wide and influential informational turn, a 
fourth revolution in the long process of reassessing humanity’s fundamental nature and role 
in the universe. We are not immobile, at the centre of the universe (Copernicus); we are not 
unnaturally distinct and different from the rest of the animal world (Darwin); and we are far 
from being entirely transparent to ourselves (Freud). We are now slowly accepting the idea 
that we might be informational organisms among many agents (Turing), inforgs not so 
dramatically different from clever, engineered artefacts, but sharing with them a global 
environment that is ultimately made of infonnation, the infosphere. The infonnation 
revolution is not about extending ourselves, but about re-interpreting who we are. When ACs 
become commodities, people will accept this new conceptual revolution with much less 
reluctance. It is humbling, but also exciting. For in view of this important evolution in our 
self-understanding, and given the sort of IT-mediated interactions that humans will 
increasingly enjoy with other agents, whether natural or synthetic, we have the unique 
opportunity to develop a new ecological approach to the whole of reality. This approach is 
not just biocentric and does not privilege only the natural or the untouched, but treats as 
authentic and genuine all forms of existence and behavior, even those based on synthetic or 
engineered artefacts. In the end, how we build, shape and regulate ecologically the new 
infosphere is the crucial challenge brought about by ACs and the fourth revolution (Floridi 
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[2007a]). Beatrice would not have understood “an artificial Companion ” as an answer to her 
question. Yet future generations will find it unproblematic. It is going to be our task to ensure 
that the transition from her question to their answer will be as ethically smooth as possible. 


Conditions for companionhood - Stephen Pulman 


Abstract 
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This chapter is an attempt to outline a set of conditions that are jointly necessary and 
sufficient for any entity, virtual or real, to be regarded as displaying the properties 
characteristic of our ordinary everyday understanding of the notion of a Companion. 


It is rather unfashionable these days to look for necessary and sufficient conditions for 
concepts (Fodor 1998). In fact, I agree with Fodor that very few non-technical concepts, 
particularly those involving a social or functional dimension, have fully satisfactory 
definitions, where by a definition we mean a set of conditions that make a biconditional of 
the form 'X is an instance of concept Y if and only conditions CL.Cn hold’ true. At most 
there are one way implications: if X is an instance of concept Y, then conditions Cl...Cn 
hold’. 

Nevertheless, we often learn something useful in the attempt to find such necessary and 
sufficient conditions, and if we are in the business of trying to build artificial Companions 
then, for practical reasons, it is important to try to pin down such conditions: in particular, 
having a set of necessary and/or sufficient conditions will tell us what we need to include in 
an implementation and what might be optional. This chapter is an attempt to outline a set of 
conditions that are jointly necessary and sufficient for any entity, virtual or real, to be 
regarded as displaying the properties characteristic of our ordinary everyday understanding 
of the notion of a Companion. 

We would normally think of the notion of a Companion as requiring some display of at least 
rudimentary cognitive abilities and some degree of intelligence. It is therefore natural to 
begin with a consideration of the role of the Turing test in defining Companionhood. 
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I would maintain that any computational system that could pass the Turing test 
(Shieber:2004) would surely be able to serve as a Companion, for that test has built into it 
the implicit requirement that the system shares enough of the same experience of the world 
as we do to be able to react appropriately to anything we say. Recall that in this test, we 
communicate with a system about anything we like (via a keyboard, say) and the system 
provides relevant and appropriate responses. If we are unable to tell whether the responses 
are provided by a human or a computer (in the original version, the choice was 
between a man and woman!) then Turing's claim is that it would be correct to describe the 
system as displaying intelligence. While the Turing test has frequently been criticised as 
inadequate as a defining measure for intelligence, this is not essential for us: provided we are 
happy to agree to the thesis that, if something passes the Turing test then it is intelligent 
(rather than if and only if), it can serve our purpose. For something that will respond 
appropriately whatever we say to it will surely be acting as a Companion. Consider the 
various things that would be involved in such a perfonnance: the system would have to 
remember details about us across a long conversation, or several conversations, have 
sufficient understanding of our likely beliefs and intentions to take them into account when 
framing responses, as well as sharing enough of our knowledge of the world to avoid 
unintentional ambiguity, to interpret our utterances correctly, and so forth. Criticisms of the 
Turing test that it is too language-centred, fail to take into account the fact well known to 
computational linguists, that it is the non-linguistic components of language understanding 
that are in many ways the most challenging. All human life is there in the 'relevant' in 
'relevant response’. 

So passing the Turing test is surely a sufficient condition for a computational entity to be 
called a Companion. But it is certainly not a necessary one: humanly intelligent behavior, 
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verbal or otherwise, cannot be a necessary condition, since people claim to get 
Companionship from cats, dogs and other creatures. Of course, people keep a wide variety of 
creatures as pets, from stick insects and spiders, snakes and mice, to sheep and horses and, 
for all I know, elephants. But not every pet counts as a Companion: I think it would be 
difficult to plausibly claim that stick insects, spiders, snakes or goldfish supply 
Companionship (although I am sure that some people do claim this). 

What makes cats and dogs and other largish mammals count as Companions is that they 
interact with us in a way that goldfish, snakes and spiders do not (in my experience, at least): 
they show affection, they respond to sounds and actions, and they have an awareness of us. 

In other words, they have intentions towards us. 

So the first necessary condition for an entity, physical or virtual, to count as a Companion is 
that: 

1. A Companion should have intentions towards us. 

This still needs some elaboration, along two dimensions. Presumably spiders have intentions 
not to get eaten or squashed, and will run away when we loom into their sight. But their 
intentions, while they involve us, are not directed towards us as individuals rather than just as 
part of the landscape. Furthermore, the intentions that Companions have towards us should 
be generally positive and benign, although this is probably not necessary. An unpleasant or 
surly Companion is still a Companion, and it is possible to imagine scenarios in which 
negative attitudes could serve a purpose. A nagging Companion might well improve the 
health of the nation. But positive or negative, a Companion should recognise us as unique 


62 



agents and be able to distinguish us from other people and animals, otherwise it could not 
have intentions towards us as opposed to, say, any nearby human. 

2. A Companion should recognise us as individuals. 

It follows from this that Companions should have good memories: recognising that you are 
dealing with the same individual across many different episodes involves a form of memory, 
and some kind of learning of your habits, since they are in part what make you an individual. 

Could static inanimate objects count as Companions? Some people get comfort from 
photographs or images of loved ones. But I think this must count as 'referred’ 
Companionship: the likeness calls the original sufficiently to mind to trigger some of the 
same feelings that interaction with the original would. 

What should a Companion look like? Curiously, it seems as if this would matter less than we 
might have expected, partly because it is possible to imagine a Companion as merely a 
disembodied voice. (Some people already have such Companions, unfortunately). Clearly, if 
a Companion has a visual representation or physical presence then for most people it 
probably needs to be something that does not disgust or frighten us. However, I can imagine 
something horrifically ugly or disgusting being a popular fonn for a Companion among 
teenage boys. In general, seeing what people do to themselves by way of tattoos and 
piercings probably makes it unwise to predict one way or another. 


If we were able to to this technologically, I can imagine people choosing the form of their 
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Companion as an expression of one side of their personality, a little like the daemons, 
typically in the fonn of animals or birds, that belong to some of the characters in Philip 
Pullman’s "His Dark Materials" trilogy. 19 (NB No relation: he spells his name wrongly.) 

So a Companion need not look much like us. I can imagine interacting perfectly well with 
something that looks like a mobile phone, a traditional robot, a Nabaztag rabbit or a furry toy 
(although I would feel silly doing so in public). I gather that we usually react better to an 
animated line drawing than a not-very-good supposedly realistic face. If something artificial 
is going to look like us, it has got to look exactly like us: a near miss can be a disaster. This is 
the 'uncanny valley’ effect Mori,MacDorman (2006), well known in the world of robotics and 
human computer interaction, whereby empathy for an artefact grows as they become more 
human, until a point just short of absolute accuracy is reached: at this point other emotions 
like fear and disgust come into play. 

The reason for the uncanny valley effect is clear: if something does not look like us, we 
come with no prior expectations about how it will behave. But if it does look very like us, we 
expect it to behave exactly like us, and our expectations set a higher standard than any 
currently existing artefact can satisfy. 

There is a computational linguistics analogue of this situation. If we are dealing with an 
interactive system that outputs English sentences, we expect it to be able to understand 
similar sentences. A good principle of natural language dialogue system design is that you 
should never output (even canned) sentences that could not be accurately interpreted, by the 
same mechanism. 
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However, even if we have a candidate for a Companion that looks exactly like us, we need 
something else too. Autistic, drunk, and mad people look exactly like us: but such people 
nevertheless are often perceived as threatening in various ways (of course, drunks are often 
threatening in a literal sense). The reason for this seems to be that such people's behavior is 
not predictable. Our interactions with them can be stressful, because we are on our guard 
more than we would be in nonnal interactions. In the typical situation we can be relaxed 
because we usually know roughly how it is going to go, that behavioral and social norms will 
be respected, and that nothing out of the way is expected from us. So it seems as if for 
something to count as a Companion, its appearance is really not very important: it can look 
like us, or not. What really matters is whether its behavior is predictable, and whether we feel 
comfortable and unthreatened in its presence. This gives us our third necessary condition: 

3. A Companion's behavior should be predictable. 

At the very least, our Companion should not violate norms of social interaction in such a way 
that would frighten or embarrass us. 

Who initiates interactions with artificial Companions? We clearly want to be able to ask 
them for help when we need it. But what about a Companion offering help? Real human 
Companions do this: mostly people do it at the right time, because they can see that we are 
struggling with some task or problem. But in general, a Companion should only interrupt us 
when absolutely necessary. The irritating behavior of the Microsoft paper-clip agent is well 
documented (http://en.wikipedia.org/wiki/Clippy). There seem to be several reasons for this 
irritation. First, the agent interrupts whether or not that is appropriate, because it is triggered 
by the application of some simple rules that monitor what you are typing. It is as if 
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someone is looking, uninvited, over your shoulder all the time - in a human, that would just 
count as rude behavior. Secondly, the facial expressions of the agent are often found 
inappropriate: smiles and jaunty little dances when you are wrestling with the peculiarities of 
Word. Thirdly, the agent appears to have no memory. It takes no account of previous 
interactions, or at least not enough account, and does not modify its behavior in the light of 
these. 

So Companions need to have a fairly elaborated and accurate model of our abilities, our 
inabilities, our interests, and our needs. This model also needs to be kept up to date and to 
keep account of previous interactions. If they do not have such a model, then artificial 
Companions are liable to display the kind of irritating, intrusive, and inappropriate behavior 
just described. But if an artificial Companion had such a model, that would mean in effect 
that our behavior would become almost as predictable to them as their behavior should be to 
us. This gives us our fourth condition: 

4. Our behavior should be predictable to a Companion. 

Whose responsibility is it for keeping the relationship with a Companion going? It is a 
common occurrence that friendships evolve from a position of equality to one where one 
individual becomes too dependent on the other. Everyone needs to put some effort into 
relationships from time to time. But relationships that need a lot of effort and input to keep 
going become a drain on one's energy, and also usually become much less rewarding for that 
reason. Unless a sense of duty or some moral obligation requires us to carry one 
sustaining them, it is usually better to move on at this point. 
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The same thing must be true for artificial Companions. A Companion should not be needy: 
we don’t want to become carers for inefficient or stupid virtual entities. They should be 
sufficiently independent of us in that they need no (or little) effort from us to keep running. 
For most of us, keeping things ticking over with family, friends, and colleagues absorbs all 
the energy we may have for such relationships. We do not want constant demands for 
attention or support from a computer program. The Tamagotchi (a craze among children 
some years ago), is a dumb pet, not a Companion. A further condition then requires a 
measure of independence and autonomy in a Companion: 

5. Companions should be independent: they should not require (much) effort from us. 


Other candidates for such conditions, not discussed here include emotion: Do Companions 
need to be capable of recognising and displaying emotion? It is commonly said that 
computer agents are more 'believable’ if they display emotion. My preliminary conclusion, 
would be that they probably need to react or recognise our displays of emotion, but need not 
display emotions themselves. Other obvious candidates are Humour, Reactivity and having a 
distinct Personality. 


No doubt there are many additions and refinements necessary to the preceding list of 
conditions for Companionhood. We may even conclude that there is no ultimately 
satisfactory set of necessary and sufficient conditions: there seldom are, in fact, for complex 
concepts. Nevertheless, trying to find them usually gives us some insight into the concept we 
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are trying to nail down. 
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Arius in cyberspace: Digital Companions and the limits of the person - 
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Abstract 

The relationship between a Companion and a person will become increasingly problematic as 
Companion technology improves and as models of users become increasingly sophisticated. 
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and the simple dichotomies that make the Turing Test so plausible as a means of determining 
intelligence will become harder to maintain. As with any kind of content-storing technology, 
such as writing, or in more recent years laptops, the amount and quality of cognition that a 
human can ‘export’ to these outside technologies is significant. The ‘person’ or ‘agent’ can 
be seen as an extended system including the technologies as well as the human, in which the 
technologies, among other things, can help in the extension of trust towards the human. 


By speech first, but far more by writing, and more again by printing, man has been able 
to put something of himself beyond death. In tradition and in books an integral part of 
the individual persists, and a part which still works and is active, for it can influence 
the minds and actions of other individuals in different places and at different times: a 
row of black marks on a page can move a man to tears, though the bones of him that 
wrote it are long ago crumbled to dust. 


Julian Huxley, The Individual in the Animal Kingdom, 1910 


Introduction 

There is a fundamental problem with the Turing Test (Turing 1950). It is designed to help 
decide whether a machine has genuine intelligence comparable to that of a human, and 
Turing asks us to imagine that it try to simulate real human performance with an interface 
preventing the use of anything other than perfonnance to distinguish human and machine. It 
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is pronounced intelligent when its performance is reliably indistinguishable from a human 


comparator. 


This has been a profoundly influential view, especially in the field of Artificial Intelligence. It 
is not obviously impossible - the Loebner Prize is awarded annually to chatbots which try to 
fool some human judges into thinking they are human, although none has yet managed to 
convince sufficiently many judges to be pronounced as having ‘passed’ the Turing Test itself. 
However, the structure of the test ignores the obvious aspect of a machine - that it is a 
machine and is not human. 


Machines are usually created to go beyond human capabilities, which is why the Turing Test 
is an unrealistic measure of machine intelligence. Why would we ever want a machine to 
imitate human intelligence exactly? Without context, there is no real measure of success. The 
Loebner Prize provides a sort of context, but a rather self-referential one; the achievement of 
convincing 30% of the judges (or whatever threshold one finds persuasive) is the end, rather 
than the beginning, of that particular technological route. On the other hand, a chatbot that 
can cope very well in a restricted context (e.g. dealing with enquiries to a bank) tends to be 
regarded as unintelligent because of the restriction which gives point to all its endeavours. In 
an obvious analogy, a trowel that passed a gardening version of the Turing Test by being 
sufficiently like a human hand for digging would be a very bad trowel - what is needed is a 
piece of steel that can dig more efficiently and without injury. The nature of the trowel as 
machine of itself removes any requirement to perfonn like a human equivalent, and this is as 
true in the production of intelligent behavior as in any other activity. 
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It is also incorrect, as I shall argue in this chapter, to take this to mean that the distinction 
between human and machine is clear and absolute. The exhibition by a machine of machine 
intelligence should be judged by standards that respect the contexts of design and use, 
standards that are likely to be very different from those we apply to human beings. 20 That 
does not mean, however, that it is trivial to draw distinctions between humans and the 
technologies we employ. Artificial cognitive support is not best understood as a separate 
adjunct to human cognition; the former strongly affects the nature of the latter. 

The issue arises when we consider the development of Companions, artificial agents which 
act as intermediaries between ourselves and a complex online world. Wilks and colleagues 
are exploring the science and social science of this possible technological future (Wilks 2005, 
2006, Benyon & Mival 2008), and predictions of how the field will develop are bound to be 
premature and speculative. In this chapter I assume the success of the research programme, 
and will explore some of the issues that may arise if Companions are routinely deployed in 
human societies. 


When I use the term ‘Companion’ I will not use the term in its usual English meaning, but 
rather refer specifically to an agent (a) with a telos detennined by the intentions of an 
individual human being, and (b) which contains a great deal of knowledge about that 
individual, obtained from the individual. That individual I shall refer to as the ‘original’. I 
will generally assume that a Companion is associated with a single original, although there is 
no particular reason why it should not be more promiscuous and associate with a range of 
originals, integrating their memories and knowledge. I will consider some of the implications 
of this, but for simplicity I will usually assume a one-to-one association. 
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Companions need not be human-like or humanoid; there is nothing about Companionship that 
demands a Turing Test sort of approach, and no reason why an Artificial Companion should 
ape the performance of human (or animal) Companions. There are many human Companions 
whose performance is lamentable even within constrained environments, such as K’s 
ludicrous assistants Arthur and Jeremias in Kafka’s The Castle whom K drives from his 
service because they cannot steer him through the castle’s bureaucracy. One feels that 
Artificial Companions might have been more useful. 


I shall take a very optimistic view of the development of Companions, to sharpen the 
philosophical debate. There is good reason, for example from robotics deployment, to believe 
that such artefacts, possibly embodied in agreeable forms such as Wilks’ suggestion of a 
furry handbag, could be popular. As Turkle argues in this volume, we will be challenged to 
see robots as “evocative objects.” Wilks’ idea is that a Companion would be an intelligent 
interlocutor 21 able to process conversational language; their owners would chat to them while 
they build up a robust model allowing them to represent the owner in complex dealings with 
fast-changing technologies. Complexity and change are among the highest of the barriers to 
entry of the Internet and the Web, and will continue to be so for many sectors of the 
community. A Companion is a relatively stable entity with which the technophobic can 
identify and bond. The Companions research programme is obviously at an early stage, but it 
is an attractive potential solution to some of the more severe digital divides. 

Autonomous agents that do one’s bidding are already technological possibilities - shopbots 
and negotiation agents are fruitful areas of current research - but not particularly challenging 
to current ideas of identity and personhood. They may affect one’s person - for example, they 
may bring on feelings of alienation or paranoia - but the current state of agent technology 
does not raise questions about what is ‘internal’ and what ‘external’ to the person. 
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However, a Companion would possess a lot of information about its original user from a first- 
person perspective. Much of its content would be unavailable from any other source. 
Depending on their skills of mimicry, and there is evidence that mimicry is an important 
support for building trust relationships. (Pentland 2008), they may even sound like their 
original or use their characteristic speech/writing patterns. These agents would contain very 
rich representations of their originals, and would share much of the background infonnation. 
In this chapter, I shall defend the idea that Companions developed to this level of 
technological advancement - particularly if they were widespread throughout a society - will 
raise issues for our understanding of identity and personhood. The way to resolve these issues 
will depend on many contingent, contextual and cultural factors, and as concepts like 
‘cognition’ expand we are never forced to apply them in a particular way, so this chapter is 
neither a prediction nor a strongly normative recipe. My main aim is to illustrate some 
interesting uncertainties in the relationship between humans, society and technology. 

The structure of the chapter is as follows. First, I shall defend the idea that a full description 
of human psychology or cognition must deal with more than the state of a person’s 
head/brain/mind (i.e. rejecting what has been called ‘methodological individualism’). 
Secondly, I shall look at issues of identity that arise from augmentation of the individual with 
technology and the creation of Companions. These two discussions will, I suggest, open up 
the possibility of some kind of internal or organic relation between a person and his or her 
Companion. Thirdly, I shall consider some of the possible types of relationship that might 
hold between a Companion and its owner. Fourthly, I shall look at some of the issues of 
authority that we can expect to arise. The chapter ends with a brief conclusion. 


Changing Boundaries 
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The methodological individualism suggesting that a person should be considered in isolation 
from his or her environment, a position perhaps held most purely by Fodor in his theory of 
‘methodological solipsism’ (Fodor 1980), has led to a number of related ideas about 
psychology and technology. It is a root of recurrent difficulties in motivating a convincing 
rebuttal to the idea that we are merely ‘brains in vats’, or otherwise persistently and 
pathologically misled about the nature of our surroundings, a sceptical problem that took on a 
new lease of life when philosophers, cognitive scientists and practitioners of Artificial 
Intelligence were able to exploit analogies between hardware/body and software/mind, and 
convinced themselves that humans were equivalent to computers which could be, and 
frequently were, misled about the outside world. 


The arguments for and against this position have been well rehearsed. What I want to argue 
in this section is that the simple analogy will appear inadequate in a world marked by wide 
deployment of effective Companion technology. In this section, I will focus on two issues 
that have impact in such a world: the increased tendency towards remoteness and the 
‘disappearance of the body’ in social interaction, and the persistent habit humans have had of 
outsourcing or supporting mental function by suitably crafting the environment. These 
undennine straightforward distinctions between mind and body, human and machine, or the 
real and the virtual. 


The Disappearance of the Body and Globalised Trust 
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The proportion of significant face-to-face contacts is falling all the time in what has been 
called by sociologists the disappearance of the body (Giddens 1990). We communicate by 
phone, email, letter, text; more and more interactions are mediated through digital 
technology. 


Identification is then a serious issue. In a series of face-to-face meetings, it is a trivial matter 
of memory to check whether someone was the same on each day; personal appearance is an 
effective biometric. One can be fooled by identical twins or a master of disguise, but face, 
voice and mannerism recognition is trivial for a human, and our society has augmented 
human evolution with other methods to deal with less familiar persons - signatures, seals and 
passwords. 


None of these standard high-bandwidth methods will work for the absent presence. Instead, 
we use technologically-constructed versions of ourselves, or avatars. The avatar must leave a 
trace behind which provides a trail back to the original person whose body has disappeared, 
so that credentials can be compared. 


Each time a new technology appears allowing people to communicate without an immediate 
physical presence, a new abstraction is created - an email log, a digital representation of non¬ 
verbal communication, a certificate of trustworthiness or whatever. As fewer demands are 
made of our physical presence, we need increasingly many of these abstractions. As the 
world has adapted to the disappearance of the body, these informational avatars have become 
ever more prominent. One often needs a bank account or credit card to be allowed to perform 
certain actions, sometimes unrelated to financial transactions. Mobile phones are developing 
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into multi-purpose trusted identifiers, and technologies such as Near Field Communication 
(NFC) will boost this trend. Gradually, technological mediation will cease to be a method of 
securing traditional interactions, and will become a necessary precondition of existing in the 
modem world (e.g. collecting government benefits, or accessing one’s bank account). 

This process of abstraction is part of a general move, characteristic of modernity, towards 
expanding the extent of trust. As I have argued elsewhere (O’Hara 2004a, 75-94), trust in 
society, the ordinary social glue that allows potentially risky interactions from which both 
sides stand to gain to take place, is generally bootstrapped by a localist model of trust in 
personal acquaintance, no doubt kick-started by a baby’s instinctive and unconditional trust 
of its parents, Baier 1994). Nevertheless, this local trust is heavy on resources for monitoring, 
and limits the extent of one’s dealings to those with whom one is personally acquainted. 
Proxies for acquaintance can emerge, such as kin relations and reputations, but even so it has 
been argued (Fukuyama 1995) that societies whose trust models are local and tribe- or kin- 
based tend to suffer large opportunity costs and are less prosperous as a result. 

Trust needs to be globalised, in order that we can trust trustworthy people with whom we are 
not acquainted. This is a complex matter, usually involving the intervention of institutions to 
certify trustworthiness and administer sanctions to wrongdoers. The disappearance of the 
body is a sign of globalised trust - tokens, including brands and logos, are used to associate a 
person (or organisation) with an interaction, thereby enabling him or her (or it) to be traced in 
the event of a betrayal. In the digital age abstractions are designed whose properties ensure 
that the interaction can be trusted by those who take part. The combination of hardware and 
software on a credit card, together with some personal knowledge such as a password or a 
PIN number, or a physical token such as a signature, fingerprint or photograph, enables a 
retailer to trust someone whom they have never met before. 
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The Companion is an example of a technology for globalising trust. It would intercede ‘for’ 
its original, and represent it in some transactions. That means that it must have a strong 
connection with its original, and would pursue the original’s interests. Haraway has urged 
that this is a cold war model of interaction between person and machine - “biological 
evolution fulfils itself in the evolution of technology” (Haraway 2004b, p299) - and argues 
that an ethics of care for the “hybrid machine-organism” is more appropriate. Hybridism is an 
issue, as we have already argued in relation to the Turing Test and the spurious distinction 
between human and machine; we discuss it in the next subsection. However, the Companion 
is best taken as a component of the hybrid, and so it makes sense to try to understand it as 
something whose purposes are bound up with those of its original. We can talk of the 
Companion as having a function, which will in some way be bound up with the interests of 
the individual, and to represent the individual in a world of globalised, anonymous trust. 


Extended Agency and the Export of Memory and Cognition 

Such abstractions are part of a general cognitive strategy used by humans to facilitate action 
in the world which has helped hone basic reasoning processes using embodiedness, 
socialisation and the environment. The basis of human reasoning, as of all higher animals, is 
that of fast pattern-matching by multiple neural systems using various none-too-intuitive 
methods which have evolved opportunistically. Certain types of message become extremely 
embedded in behavior and instinct, as cognitive structures and social patterns co-evolve. 
Many animals modify their environment, for example, by marking their territory with a scent; 
socialised animals of the same species then become extremely good at recognising and 
identifying the scent. In humans face and expression recognition is particularly sensitive, as 
they are used for communication both conscious and unconscious. In each case, the ability to 
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process particular aspects of reality is heightened in order to aid socialisation and 
communication. The work of adapting the environment can be outsourced to organisms of 
other species who co-evolve with us, ranging from intestinal flora to highly socialised 
animals such as dogs or horses (Haraway 2004b). 


Clark has argued that the hallmark of specifically human cognition is not found in 
fundamentally neural processes, but rather in “our amazing capacities to create and maintain 
a variety of special external structures (symbolic and social-institutional). These external 
structures function so as to complement our individual cognitive profiles and to diffuse 
human reason across wider and wider social and physical networks whose collective 
computations exhibit their own special dynamics and properties” (Clark 1997, p 179). 
Augmenting theories of extended cognition have come accounts of extended agency and 
extended (moral) responsibility (Hanson 2009). 


Substantial problem-solving work is offloaded to the environment, which is shaped by human 
action (including interpreting some objects as symbols) in order to act as either a corporate 
memory or as a short cut in the problem-solving process. This shaping/restriction of context 
substantially reduces cognitive load (O’Hara & Shadbolt 1997); the more aspects of the 
environment that can be assumed, the less infonnation processing needs to be done, because 
facts about the environment can be asserted rather than derived. To use Clark’s term, 
reasoning is diffused across the environment by human action, allowing us to succeed by 
using our intelligence more intelligently. An alternative way of looking at this is to see the 
mind as the human brain plus the engineered environment; the engineering of the 
environment in effect allows us to include some reasoning as common property. 
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When the engineered environment includes virtual representations, the physical, the digital 
and the cognitive become harder to separate out, as for example with military representations 
of battlefields presented as augmented perceptions to soldiers. One ambitious gadget is a 
helmet whose visor provides a transparent display mapped onto the view through it, showing 
street and landmark names, identifying what is out of the line of sight and identifying threats 
such as sniper positions established by reconnaissance (Bulman et al 2006, p.p 119-122). In 
such an environment, a central example of what Turkle in this volume calls “our culture of 
simulation”, it would be just as much a mistake to postulate a strict separation of external 
reality and cyberspace as it would be to assume that real life was just another window on the 
visor (Zizek 1997, p. 132). The two ‘realities’ are intertwined, li nk ed by the cognition of the 
soldier. 


Theories of individualistic rational mind are hard to justify in the abstract precisely because 
of the complexity of inferring everything from scratch about the context, but when choices 
are severely constrained they look much more plausible (Satz & Ferejohn, 1994). Reasoning, 
on this reading, becomes a communal activity, even when the locus of activity appears to be 
the individual mind. As David Hume reminds us, all human action depends absolutely 
crucially on interaction with others, whether we acknowledge it or not. 


The mutual dependence of men is so great, in all societies, that scarce any human 
action is entirely complete in itself, or is performed without some reference to the 
actions of others, which are requisite to make it answer fully the intention of the agent. 
The poorest artificer, who labours alone, expects at least the protection of the 
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magistrate, to ensure him the enjoyment of the fruits of his labour. He also expects, 
that, when he carries his goods to market, and offers them at a reasonable price, he 
shall find purchasers; and shall be able, by the money he acquires, to engage others to 
supply him with those commodities, which are requisite for his subsistence. In 
proportion as men extend their dealings, and render their intercourse with others more 
complicated, they always comprehend, in their schemes of life, a greater variety of 
voluntary actions, which they expect, from the proper motives, to co-operate with their 
own. ...A manufacturer reckons upon the labour of his servants, for the execution of 
any work, as much as upon the tools, which he employs, and would be equally 
surprised, were his expectations disappointed. In short, this experimental inference and 
reasoning concerning the actions of others enters so much into human life, that no 
man, while awake, is ever a moment without employing it. 

(Hume 2007, pp.64-5) 


Trust in the reliability of others makes human society possible (Seabright 2004), and indeed 
without it it is hard to see how any kind of moral relationships could even be definable (Baier 
1997). That reliability extends to interpretations of functional objects - for instance, money, a 
vital store of value without which restructuring of the environment would remain 
rudimentary, and a long way short of the sophistication of modern urban environments where 
so much thinking is, as it were, done for us in concrete. 


The symbolic significance of our alteration of the environment varies from the basic (e.g. 
flattening floors to make walking easier) to highly complex intennediaries such as money or 
language. Digital technology gives an extra dimension to cognitive scaffolding. It allows the 
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creation of artefacts, such as Wilks’ Companions, which can act independently of human 
control. Of course, the actions of artificial agents require a human-created and mediated 
environment (as do human actions), but they do not need direct command. 

Conversation and Independence 

Independence does not need digital technology to support it. As Julian Huxley reminded us at 
the beginning of this chapter, all sorts of communicational methods have been used to 
socialise or outsource mental function. Most of these have been fiercely resisted, because of 
fears of a decline in that mental function, although it often turns out that other functions can 
be enhanced given the lightened cognitive load. The dilemma was put in a particularly clear 
form by Plato, through the mouth of Socrates, in the Phaedrus. 


You know, Phaedrus, writing shares a strange feature with painting. The offsprings of 
painting stand there as if they are alive, but if anyone asks them anything, they remain 
most solemnly silent. The same is true of written words. You’d think they were speaking 
as if they had some understanding, but if you question anything that has been said 
because you want to learn more, it continues to signify just that very same thing 
forever. When it has once been written down, every discourse roams about everywhere, 
reaching indiscriminately those with understanding no less than those who have no 
business with it, and it doesn’t know to whom it should speak and to whom it should 
not. And when it is faulted and attacked unfairly, it always needs its father’s support; 
alone it can neither defend itself nor come to its own support. 

(Plato 1997a, 552 [275de]) 
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This is a fascinating and ironic passage. Socrates, of course, did not write anything down and 
may well have actually held views not unlike these. On the other hand, his immortality is 
guaranteed by the written accounts of him by Plato, Aristotle, Xenophon and Aristophanes. 
His point in the Phaedrus applies just not only to relatively primitive technology such as 
writing, but also to more involved digital technologies such as Companions. His reservations 
can be summarised as the following points. 

1. Writings cannot explain themselves. 

2. The circulation of writings cannot be restricted. 

3. Writings cannot defend themselves against attack. 

Successful written communication is grounded by oral communication, argues Plato (a 
written response to a written attack would be recursively vulnerable to his original argument 
once more). For these three enumerated reasons, an author loses control of his or her output. 
Only through engaged conversation does wisdom follow. Writing can only “remind those 
who already know what the writing is about.” It has no independent life. More abstractly, 
Plato grounds interaction in physical presence, and is worried by absence (such as the 
disappearance of the body described at the beginning of this section). 

As Havelock pointed out (Havelock 1963), Plato, presumably like the historical Socrates, was 
wrestling with the opportunities and threats of a new literate world. Other types of cognitive 
scaffolding as described by Clark have similar properties to writing, and are also 
unresponsive to physical presence. From Clark’s perspective, this is not a serious problem - 
there is no requirement for a cognitive support to “know to whom it should speak”, and no 
reason why it should do more than remain “solemnly silent.” It need not be able to act 
independently. 
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New technologies are beginning to change this landscape (O’Hara 2004b). In particular, 
technologies that can exploit the metadata attached to information can be more responsive by 
managing permissions and allowing reasoning about policies to protect privacy etc (Weitzner 
et al 2005), by gathering or collecting supporting information (Blythe & Gil 2004), or by 
tailoring publishing strategies to the particular requirements of the reader (Weal et al 2007). 
The reader can also bring his or her own ideas and intellectual structures to the interaction, 
using ontology’s to mediate and possibly to uncover content in the discourse that was 
unknown to the writer (Fensel 2004). 


The Linked Data Web, more commonly if less accurately known as the Semantic Web 
(Shadbolt et al 2006), might supply this revolution single-handed if it takes off as hoped, but 
even if not, personalising technologies could allow surfing the Web to seem more like a 
conversation than hitherto, on an ad hoc basis. The movement loosely known as Web 2.0 has 
shifted the balance away from writers and towards readers, with the blogosphere already 
looking like a conversation, and with aggregation techniques allowing us to see the ‘wisdom 
of crowds’ (Surowiecki 2004) giving concrete form to a General Will of the type that 
Rousseau described, very separate from the wills, wisdom or intelligence of its components. 
There is not a great deal of automation in the Web 2.0 space, but the difference from what we 
might call the ‘traditional’ Web (if something can be both traditional and under two decades 
old) is clear. 


As such technologies become more effective and commonplace there is clearly a place for 
Companions, assuming that the technology they require continues to develop. The lack of 
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physical presence of a human being is already, as I have argued, less of a handicap than Plato 
anticipated, and many of the shortcomings of the disappearance of the body are being 
addressed by technology. Artefacts can defend themselves against attack, can restrict their 
audiences, and can explain themselves. Companion technology is part of a general movement 
in this direction. 


To summarise this section, we have argued that traditional distinctions are being broken 
down by social and technological development. Brain, body and mind are strongly linked by 
causal ties, and it is a characteristic of human intelligence to alter the environment - which 
includes intellectual and virtual environments as well as the physical world - to support 
cognition and provide intellectual resources. Mind is embodied, but not just in the brain. 
Furthermore, these resources, being reusable, link minds and persons. Social development, in 
the fonn of increased and more widespread trust, is also facilitated. 

It may well not be long before stores of digital infonnation begin to demonstrate 
independence of the minds that created them, not only in terms of their content (being 
assembled by a number of people and thereby transcending the knowledge or memory of 
each individual), but also in terms of action. In that case, the extended agency position as 
sketched for example by Clark would be significantly complicated. In the next section, we 
will consider some of the implications. 


Augmentation and Companionship 
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Centralisation of intelligence has been a common assumption throughout the prehistory and 
history of cognitive science. Most works in the early philosophy of Artificial Intelligence 
assumed that a single controlling executive would be required in order to produce intelligent 
behavior artificially. The so-called ‘good old-fashioned AI’ or GOFAI programme whose 
individual successes never added up to the achievement of the goal. The idea of distributed 
intelligence came later, and also provided some success, but the general assumption remained 
that Artificial Intelligence would be realised in standalone devices until the World Wide Web 
came along. Even then the successes of statistical techniques such as PageRank surprised 
many of the major figures in the field. The Semantic Web still struggles to evade the 
accusation of adopting the same fallacies as GOFAI (Sparck Jones 2004, Wilks & Brewster 
2006), although more recent discussions of the SW emphasise the importance of local 
ontologies, small projects and avoiding overweening ambition (Berners-Lee et al 2006, 
Shadbolt et al 2006, Alani et al 2008). 


Meanwhile, other types of web presence have mutated from individuals to less well-defined 
collectives. The lifelong, a comprehensive and indiscriminate record of someone’s existence, 
has typically been seen as being strongly associated with a particular person but the arrival of 
social networking has resulted in lifelogs with distributed properties (O’Hara et al 2009). 
These developments have mirrored those in psychology which dethroned central executive 
function (Libet 1985, Sheth et al 2009). 


Even commentators who are sensitive to technology’s transgressive tendencies can find it 
difficult to break out of this type of methodological individualism. For instance, Haraway in 
her work on cyborg culture argues that “the dichotomies between mind and body, animal and 
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human, organism and machine, public and private, nature and culture, men and women, 
primitive and civilized are all in question ideologically” (Haraway 2004a, 22), and that in 
particular the organism/machine distinction is particularly “leaky” (Haraway 2004a, 10-11). 
She writes at length about the possibilities that unfold for liberating human identity when 
machine prosthetics become available, and about her “dream not of a common [and therefore 
totalitarian] language, but of a powerful infidel heteroglossia” (Haraway 2004a, 39). 
Nevertheless, she still places her self, and those of others to be liberated, on the same level as 
the cyborgs whose imperatives and Cold War myths she wishes to rewrite. 


Similarly, her work on Companions focuses on dogs, which “are not a projection, nor the 
realization of an intention, nor the telos of anything” (Haraway 2004b, 300). This is of course 
an important point to make about dogs (and other Companion animals, including humans), 
but although she eloquently unpicks the ethical imperatives that follow from the co-evolution 
of dogs and people, she nowhere follows the thought that in the realm of the artificial, telos is 
an important factor. In a world where Wilks-style Companions are commonplace, the ethics 
of interacting with teleologically-specified agents may be more complex. We should not be 
surprised if Companions evolved from standalone applications to more complex entities with 
complex functions; in this section I will sketch some potential implications. 


Extensions in Time 
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As Wilks has often emphasised, the Companion, being artificial, may well outlive the original 
person from whom it gained its content. There is no reason why in such circumstances the 
Companion could not continue to perform its function in society as an intermediary for some 
aspect of the digital world. It may well even be that over time the number of such 
Companions multiplies to create a substantial population, as for example in Charles Stress’ 
novel Saturn’s Children, in which the dwindling human race has created a whole range of 
types of robot designed to do its bidding, from labouring drones to concubine sex machines. 
After humans finally die out, the robots carry on following their programmed imperatives, 
but with enough intelligence to continue within the skeletal framework of the human society, 
forever trying to please humans who are now mythical (“the sad fact is, human civilization 
did not even break for lunch when humankind died out.”). 


A sufficiently intelligent Companion could function as an important memorial or source of 
information about the original, now deceased, person. Great-grandchildren could question the 
Companions of their great-grandparents in order to build up their knowledge of barely- 
remembered figures. If the Companion was a repository of the preferences of the original 
person (for example in legal matters), then it could infer the original’s likely wishes in 
complex cases involving disputed wills, the management of trusts, or legal documents whose 
measures were now out of date, for example, assuming the Companion’s knowledge base was 
reliable and properly certified as containing an accurate reflection of the original’s desires. 
Rather than search for a living will, a doctor might consult a Companion to try to determine a 
preference as to whether an apparently brain-dead person would wish her life support systems 
to be switched off. In the even that the support systems remained switched on, the 
Companion might continue to act for her. Such a Companion would enjoy a special 
relationship to the original person. Important projects could continue, and the wisdom of 
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older generations preserved and consulted. That is not to underestimate the difficulties of 
providing what C.P. Snow memorably called “that singular necrophilic confidence ... about 
what a dead relative would have ‘liked,but rather to expect a sufficiently intelligent agent 
with a sufficiently good model of the original person in a sufficiently well-specified and 
circumscribed context to be reliable, and to be certified to be so. 


Federations of Identity/Identities 

At least since Locke, modern philosophers have used the body as an anchor for identity, 
because - it so happens - it gives spatiotemporal continuity, hosts the sensory organs and the 
brain, and also provides the limbs that are used initially if not exclusively to investigate, 
explore and map the world. There is a debate as to whether the concept of personal identity 
requires the body (often carried out using thought experiments about interrupting 
spatiotemporal continuity or transferring of someone’s brain into a different body, for 
example), but it is at a minimum regarded as a useful proxy for the locus of identity. 
Nevertheless, when cognition or memory are exported to the environment, deliberate damage 
to that environment may be morally more significant than mere vandalism, and looks more 
like harm to a person. One does not have to hold a Marxian view of the relation between 
labour and identity to feel that the theft of a laptop computer with several years’ work stored 
on it is somewhat more like an attack on the person than, say, the theft of an equally valuable 
DVD player or a more valuable piece of jewellery (Clark 1997, p. 215). 


Identity has always been an issue in cyberspace, captured by the immortal New Yorker 
cartoon whose caption read “on the Internet, nobody knows you’re a dog.” Similarly, nobody 
knows you are a Companion. Even if we don’t go as far as those envisaged by Turkle (1995), 
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who argue that the Internet gives us unequivocal examples of the ‘decentred subject,’ the case 
of the Companion is still difficult to describe; it acts for, but is independent of, an original. 
Orders could be countennanded, but equally the Companion does have the special 
relationship to the original as postulated in the previous section. There would be no reason in 
principle why the Companion could not generate and sign legal or financial documents 
(although in practice of course society may refuse to let it happen), thereby even going 
beyond the laptop as an extension to the person. Machines already make decisions of great 
moral import, for example in the medical domain. (Tuffs 1996). 


A Companion would have an important connection to its original. Even given the original 
person was consciously aware of its artificial nature, it would still be a significant 
relationship. Ties of kinship, intimacy and similarity tend to undermine or undercut more 
formal ties, as for example in Joseph Conrad’s short story ‘The Secret Sharer’, where the 
captain of a ship shelters a fleeing murderer for whom he feels the affinity of similarity, and 
as a result neglects his duties and risks his ship. This basic relation can be complicated, 
because different Companions of the same original could be merged in various ways, with the 
potential for sideways growth of identities, again as envisaged by Stross in Saturn’s Children 
(“I haven’t worn her long enough to receive more than a perfumed hint of her presence in my 
head,” as one robotic character comments). Such relationships would be relatively 
straightforward compared with a merge of Companions from two different originals, or in the 
simpler case where a Companion’s data is deliberately falsified for evil purposes. 

Falsification of data may even be voluntary: In the context of lifelogging, Dodge & Kitchin 
(2007) suggest that privacy can be preserved for a lifelogger if he or she deliberately falsifies 
a small proportion of the lifelog, so that no-one could be sure that infonnation extracted from 
the lifelog was veridical. 
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The Companion could be equipped with inferential capacities that take the expressed 
preferences of the original and develop them in unanticipated directions. It may treat the 
original’s preferences more seriously and with more consistency (for instance, it might be 
more protective of the original’s privacy than the original himself). It could be equipped to 
take infonnation from the context or environment to amend information (for instance, 
‘correcting’ memories that it discovers are false, augmenting its knowledge base via Google 
or interacting and sharing information with other Companions). It could use the original 
person’s social network to make itself into a more communal resource. 

In short, the Companion need not be restricted to a mirror of the original. Context, online 
information resources, pre-programmed information, other Companions and other members 
of the original’s social network could also help inform its knowledge base (as well, of course, 
as its own experience). The relationship between the identities of the Companion and the 
original need not be simple or straightforward. 


Arius 2.0 

Arius was a Berber priest who died in AD336, who thought deeply about the relationship 
between the three manifestations of God - God the Father, Jesus the Son, and the Holy Spirit. 
Arianism was deeply controversial, disputed for many years in the Late Roman Empire, 
finally being denounced as heretical. 


Arius’ problem was a simple one - how to understand the Trinity. The now-official doctrine 
of most large Christian denominations which won the ancient theological and political 
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argument was that God is a unity consisting of three persons, and, of course, this is something 
of a mystery for worshippers. Arms’ idea was a simpler one, which is perhaps why it 
appealed to barbarian converts from paganism: God is three persons. God was not the Father 
before He begot the Son. Jesus, though divine, was not of the same substance as God, was a 
created being, and may even be inferior to God. 


These weighty issues are somewhat beyond the scope of this chapter, but like Arius of old, if 
the Companions idea is as successful as its advocates hope, we will be faced with a similar 
knotty problem of identity. Instead of the Holy Trinity, we will have to grapple with a rather 
more Earthbound set of body, avatar and furry handbag. 


The body of the original person is the aspect of the new trinity with which we are familiar, 
the provider of spatiotemporal continuity and the Lockean ground of identity. The virtual 
Companion is an avatar. Thirdly, there will be bodily incarnations of an avatar, whose nature 
would of course strongly affect the sorts of information it was able to extract from a person. 
Wilks playfully imagines a furry handbag to which a person chatters all day. Wilks’ own 
Companions project 22 has experimented with both embodied Nabaztag rabbits and virtual 
forms, including an animated version of the British comedian Ken Dodd. 23 There is some 
evidence that elderly Japanese people are quite happy to interact with anthropomorphic 
robots, and indeed often prefer them to non-Japanese people who cannot be programmed to 
avoid behavioral taboos. It has been suggested (Chrisley 2003) that the Kismet robot 
developed at M.I.T. 24 is so compelling as an interlocutor because it has extremely expressive 
eyebrows. The form the ‘furry handbag’ takes will be very important for the type of 
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information that can be brought out of the human and used to populate the Companion’s 


memory. 


Arius’ solution was to treat each aspect of the Trinity as different and separable; the 
victorious Athanasian Creed treats the Trinity as three aspects of one substance. Neither 
solution solves the issues with respect to Companions. The independence of the Companion 
makes it hard to regard it as the same as the original person, or merely a tool used by that 
person, while its strong links with the original make a story of separation implausible too. 


Types of Relationship 

It is usual to see human beings as taking a clearly identifiable place within a hierarchy or 
partial order of extended or augmented agency. An extended agent contains people as proper 
components. Hanson (2009) gives the example of two people colluding in a murder using a 
car, and argues that moral responsibility falls to the extended agent two-people-and-car. 
Though perhaps few would find this reasoning convincing, the point is that he views the 
individuals as components of the more complex extended agent. Even those less Catholic 
about what counts as an extended agent take an analogous view; a cyborg as described by 
Haraway, or a scaffolded mind described by Clark are basically human agents with reference 
to whom cognitive enhancements (whether machinery or amended aspects of the 
environment) are identified. Although Clark insists that a description of a person’s mind 
requires a description of the environment, he does not directly address the question of 
federation of identity. Though questions like “where does the user end and the tool begin?” 
are “a delicate call”, he does not dwell on the implications of the thought that “much of what 
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we commonly identify as our mental capacities make ... turn out to be properties of the 
wider, environmentally extended systems of which human brains are just one (important) part 
(Clark 1997, pp. 214-5). Are there any grounds to keep the individual human brains distinct? 
Does Clark’s citation of human language as a tool bring other minds into important contact 
with our own: For example using Wittgenstein’s argument that language is essentially public? 
Similarly Haraway’s Companions are teleologically separate, agents in their own right. 


Of course, the furry handbag itself (whatever fonn it takes) is likely to be physically separate 
from the body of the person, 2 ^ but the world-involving nature of human cognition, and the 
human-involving nature of artificial cognition together mean that the hierarchy of 
Companionate agency might well be much more complex than these examples suggest. The 
Companion need not simply be an autobiographical (= solely the responsibility of the original 
person) recreation of the original. As noted above, the Companion’s knowledge base might 
go beyond the infonnation provided by the original; this indeed is likely, as one of the main 
suggested ideas behind Companions is to act as an intermediary between non-techie persons 
and the complex online world. In any case, it is hard for anyone to control the picture they 
present of themselves. For instance, Dr. Johnson provided much solid input to Boswell’s 
depiction of him, and exercised a good deal of editorial control over the process (as Boswell 
admits in his Journal of a Tour to the Hebrides ), but even so Boswell determined the final 
product, distilled from conversations with Johnson and others, including testimony from 
others who knew Johnson and objective sources of infonnation where possible. The result is 
somewhat mythical, and certainly not what anyone would call an objective account, but 
equally not one that Johnson would have endorsed despite his level of input (he did not like 
being called ‘Dr. Johnson’, for example). 
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As Haraway argues, Companions are separate entities in their own right, but this is not true of 
the mechanistic Companions discussed in this chapter which derive important aspects of their 
meaning from their originals. We may ultimately have to think deeply about the ethical 
treatment of Companions (as Turkle suggests in the introduction to this volume), but at least 
some aspects of their telos will be strongly, intimately and intrinsically related to its original. 
The considerations of the previous paragraph cannot be used as arguments for completely 
detaching the agency of the Companion from the agency of the original. 


There are various examples in literature and philosophy of such complex relationships 
between original and creation, some already mentioned. In Stross’ Saturn’s Children, 
‘families’ of robots are able to share memories, experience and capacities, which though it 
does not reduce their agency to a single entity, equally it is sometimes threatening to their 
individuality. Plato’s representations of his mentor Socrates seem to have evolved from a 
lifelike portrayal of the sceptical gadfly and social irritant, with a resemblance to 
Aristophanes’ more waspish picture in The Clouds, to a more independent figure, first 
applying Socratic methods to philosophical problems without coming to direct conclusions, 
then taking a stronger, less sceptical stand on matters such as justice (e.g. in The Republic ), 
and ending up finally as a mouthpiece for Plato’s mature and complex philosophy. 26 The later 
dialogues use reasoning methods with which Socrates might have been sympathetic, but 
which appear to have developed after his death (such as the inductive method of studying the 
properties of objects in order to discover what they have in common), and one cannot help 
thinking that the real Socrates would have made great sceptical play with some of Plato’s 
mature opinions. 
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Nevertheless, the written Socrates is importantly constrained by the original; to the extent, as 
we have seen, of paradoxically arguing against writing as a means of preserving 
philosophical argument. The newly literate world in which Socrates and Plato found 
themselves was as challenging to its inhabitants as the digital world is to us (Havelock 1963), 
and Plato’s use of Socrates as a long-running character, living on for maybe half a century 
after taking hemlock, must have been a literary and philosophical conceit of impressive 
originality. In the Second Letter, the author, supposedly Plato, tells Dionysius that “there is 
no writing of Plato’s, nor will there ever be; those that are now called so come from an 
idealized and youthful 27 Socrates” (Plato 1997b, 1639 [314c]). Most scholars believe that 
Plato’s Second Letter is a forgery, but the point still stands that a near-contemporary was 
prepared to impute agency to the Platonic Socrates, however metaphorically. 


Some of the more ambitious Twentieth Century attempts to explore the limits of character 
and fiction also provide fertile ground. Norman Mailer’s Ancient Evenings circles around a 
series of reincarnated characters called Menenhetet, while Mailer tried, and failed, to produce 
an eight-volume series of novels on social and philosophical themes all featuring a romancing 
fake Irish adventurer and observer called Sergius, or sometimes Cassius O’Shaugnessy. 28 
Malcolm Lowry also played with a series of interlinked and recognisably similar lead 
characters in a series of novels (only one of which was completed at the time of his death) 
portraying a spiritual journey from Hell through Purgatory to Heaven. These characters were 
lightly fictionalised aspects of Lowry himself, and their misadventures closely mirrored his 
own. They are of course fictional, but independent critical treatment of any of them would be 
inappropriate given the nature of Lowry’s oeuvre. In these cases, the fictional characters can 
even be seen without too much hyperbole as ‘Companions’ in Wilks’ sense, animated by 
prose, who pilot their authors through philosophical, political and spiritual malaises. 
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The relationship between a person and his or her Companions is a complex one to describe. 
When the Companion is adding substantially to the original’s cognition, carrying out 
important cognitive or social functions, it is extremely plausible to see it as making a 
contribution to the identity or the personhood of the original, as would be argued by Clark or 
Hanson, for instance. When the Companion becomes more intelligent, capable and 
autonomous, then it might also be seen at the same time as having an identity or personhood 
(however derivative, metaphorical or incomplete) of its own. The simple hierarchies that are 
usually taken to characterise relations of identity between people and artificial agents become 
harder to sustain, and the relationship between Companion and original will be more tangled, 
interwoven and context dependent. 


Authoritative Representations/Representatives 

The aim of the Companion is to perform some task(s) for the original person, which it will do 
using inbuilt designed capabilities informed by knowledge absorbed from the original person 
via knowledge acquisition procedures and conversational extraction techniques. Such tasks 
might include negotiating online prices of goods, organising photographic records, 
constructing biographies to order, and so on. To perform those tasks in the stead of the 
original, the Companion must have some kind of authorisation which will presumably usually 
be provided directly by the original person using a secure certificate, but this will not always 
be necessary or (for example in the case of the use of a Companion to sort out a posthumous 
legal tangle) possible. The context in which the Companion is used to represent a person will 
detennine whether it has the ability properly so to do. 
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Assuming that the use of a Companion in a context is acceptable, the key factor then is 
whether its model or representation of the original person is sufficiently accurate. In the 
offline world, the usefulness or acceptability of a representative, such as a solicitor, will 
depend crucially on how well he is briefed about his client, and an analogous issue arises for 
the Companion. Is the model that the Companion holds ‘truthful’? 


The problem is that, where personality and opinions are concerned, the notions of accuracy 
and truthfulness are hard to pin down. The methods used by the Companion for knowledge 
acquisition and modelling will certainly be a factor. The emphasis on methodology is not 
unusual - for example, in psychoanalysis, the methods for understanding the patient’s 
subconscious are assumed to be superior to the patient’s own assessment of his situation (the 
subconscious is deemed to be more truthful than the conscious personality). 


In the virtual world, consciously-constructed avatars can be seen as more authoritative than 
the public face of the original person, even by the original herself. One can, in constructing 
an avatar, enjoy the distance and the relative lack of risk by exaggerating one’s personality, 
pretending to be brave or beautiful or psychopathic or ruthless, or alternatively, one can 
construct a persona that is ‘more’ oneself than the compromised self of everyday existence. 
One can express aspects of oneself that one would normally bury in order to keep one’s 
complex offline social life together. These two ways of relating to one’s avatar are 
intertwined (Zizek 1997, 137-8). Inhibition and shame are (usually) absent from online 
existence, and one can give voice to the hidden aspects of one’s personality as long as one is 
aware of the lack of consequence. But, as Zizek argues, it would be a mistake to focus on the 
‘truth’ about these matters. 
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When a man who, in his real life social contacts, is quiet and bashful, adopts an angry, 
aggressive persona in virtual reality, one can say that he thereby expresses the 
repressed side of himself, a publicly non-acknowledged aspect of his ‘true personality ’ 
- that his ‘electronic id is here given wing however, one can also claim that he is a 
weak subject fantasizing about more aggressive behavior in order to avoid confronting 
his real life weakness and cowardice. (Zizek 1997, pp. 137-8) 


As Goffman argued decades ago (1959), our negotiation of everyday life is akin to playing a 
series of roles as an actor, and one needs a backstage to express certain aspects of one’s 
personality that cannot be shown in public. For instance, in a working context: 


The backstage language consists of reciprocal first-naming, co-operative decision¬ 
making, profanity, open sexual remarks, elaborate griping, smoking, rough informal 
dress, “sloppy ” sitting and standing posture, use of dialect or substandard speech, 
mumbling and shouting, playful aggression and “kidding, ” inconsiderateness for the 
other in minor but potentially symbolic acts, minor physical self-involvements such as 
humming, whistling, chewing, nibbling, belching and flatulence. (Goffman 1959 p. 
128). 


Of course, this is only backstage with respect to the interaction with the public or with ‘the 
boss’; this ‘backstage’ is ‘onstage’ with respect to colleagues, and for that interaction another 
‘backstage’ is needed (perhaps the family, or friends in the pub). This social management of 
emotions has been taken online with virtual reality acting as yet another ‘backstage’ resource. 
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Zizek writes about constructed avatars which the original person both constructs and ‘acts 
through’, a personality which the person adopts, for example in an online game such as 
Second Life or World ofWarcraft, but the Companion takes the externalisation further by 
automating behavior and ultimately acting autonomously. The original person plays no part 
in the actions of the Companion, though he may help ‘steer’ it through interaction, 
conversation and so on. This distancing sharpens the question of authority, or, as Turkle puts 
it in this volume, the “notion of authenticity” becomes “threat and obsession, taboo and 
fascination”. The culture of simulation is tested at its limits; the question is not, as in the 
Loebner Prize, how to fool a group of judges in a series of artificial tests, but rather how to 
operate successfully in society. There is no Turing Test requirement to masquerade as a 
‘genuine’ intelligent being - the Companion need not and should not try to conceal what is 
most germane about itself, that it has particular capacities programmed into it. Indeed, if it 
were embodied in the form of a furry handbag, it would be very unlikely to succeed. 


The question instead is whether the Companion can actually perform genuine functions in a 
complex, dynamic social world, consistent with its teleological requirement to represent or 
intermediate for the original person. That will require a set of person-centred knowledge 
acquisition methods, and set of world-centred procedures and knowledge bases. Criteria for 
success will vary, but the closer the Companion comes to making a genuine contribution to 
our personhood, the more pressing this question will be. The Companion will need a model 
of the original person that ‘fits’ very well, and to be able to deploy that model in a complex 
world. This notion of ‘fit’ between model and original is the crux of the matter; the better the 
fit, the wider the Companion’s legitimate and authoritative deployment can be, and the wider 
the legitimate deployment, the more influence on the original’s personhood it is likely to 
have. 
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Conclusion 


If Companion technology becomes so sophisticated that the Companion is capable of acting 
independently in its original’s interests, then the relationship between original and 
Companion becomes more difficult to describe, because there will be a case for regarding the 
Companion as a person or an agent too. Much will depend on the context, circumstances and 
assumptions made at the time, particularly with regard to the trust relations - as Wittgenstein 
reminds us, there is nothing in psychological concepts such as ‘person’ that tells us how we 
should go on in new circumstances or language games. In this chapter, I have tried to produce 
a preliminary sketch of some of the considerations that might be brought in, and to indicate 
appropriate background theories that will illuminate the new situation. Nevertheless, the main 
detennining factors are the technologies themselves, and the social practices within which 
they function. 
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Section Three - Social and Psychological Issues: What should a Companion 

be like? 
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Conversationalists and Confidants - Margaret A Boden 


Abstract 

If there were an abstract for this very short paper, it would be this: "Conversationalists, 
maybe — but confidants ?" 


I assume that computer Companions (CCs) could hold conversations, of a sort, with human 
beings (HBs). Their use of language will be crude compared with that of (most) HB users. 
But they will be able to broaden and enrich the conversation not only by providing facts and 
information but also by using associative memories (calling sometimes on the user’s 
autobiographical details and sometimes also on the Internet) to say "Oh, that reminds me of 

." etc. Whether they will always avoid an ELIZA-like idiocy in response to the 

HB’sremarks is another matter: I suspect not. 


I assume, too, that some CCs will be able to exploit a very wide range of recent advances in 
AI and VR. These include, for example, being able to speak a natural language such as 
English with an appropriate local accent (Fitt and Isard 1999; Fitt 2003); being able to “lip- 
read” to aid their understanding: only sixteen lip-positions are needed to represent English 
speech (Lucena et al. 2002); and being able to detect, and seemingly to express, basic 
emotional states. CCs equipped with attractive humanlike ‘faces' might even be able to adjust 
them to suit the female user’s time-of-the-month (Penton-Voak et al. 1999). Other 
potentially relevant advances in robotics and VR are listed in Boden 2006: 13.vi.d.). 
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I also assume that, as predicted long ago (Frude 1983), people will be having sex with robots, 
and with VR systems, in various ways: indeed, this has already started to happen. Some HBs 
may even become personally obsessed with, or even infatuated by these 21st-century sex 
toys. But having sex with CCs, with or without the obsessional aspect, is very different from 
being in love with them, still less engaging in a relationship of personal love (Fisher 1990). 

So I have grave reservations about talk of ‘love ’ between HBs and computers (cf. Levy 
2007a). 


What’s much more problematic than using CCs as conversationalists, or even as sex objects 
(sic), is whether they can be, or should be used in lieu of, confidants. 

A confidant (as opposed to a conversationalist) is someone to whom one relays personal, 
sometimes highly sensitive, infonnation and emotions in the expectation of empathy and 
sympathy, in response. Sympathetic words could be produced by the CC (sometimes, in 
appropriate circumstances), but these are not enough. Do we really think that a CC could 
offer us genuine sympathy? 


Often, even in counselling situations, what the HB wants is an acknowledgment that they 
have suffered, persevered, survived... etc. The acknowledgment in question here rests on a 
shared understanding of the human condition. Do we really think that it could be offered by a 
CC? 


A "secret" is something that one HB wants to keep from some/all other HBs. To tell an HB a 
secret is to trust them (to assume that they will not spread it to other HBs), and to compliment 
them by making them a confidant in this way. 
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One can hardly ‘ compliment ’ a CC by telling it a secret. So the sense in which one might tell 
one’s secrets to a CC is very different from that in which one can tell them to a fellow HB. 


Most secrets are secret from some HBs but not others. If two CCs were to share their HB- 
users’ secrets with each other, how would they know which other CCs (i.e. potential users) to 

’ trust' in this way? The HB could of course say "This is not to be told to Tommy".but 

usually we regard it as obvious that our confidant (sic) knows what should not be told to 
Tommy — either to avoid upsetting Tommy, or to avoid upsetting the original HB. How is a 
CC to emulate that? 


The HB could certainly say "Tell this to no-one" — where ‘ no-one ’ includes other CCs. But 
would the HB always remember to do that? 


How could a secret-sharing CC deal with family feuds? Some family websites have special 
functionalities to deal with this. E.g. Robbie is never shown input posted by Billie. Could 
similar, or more subtle, functionalities be given to CCs? 

These problems would arise repeatedly if the CC enabled the HB to generate a personal 
memoir, or autobiography. The more the CC were seen as a "confidant", the more sensitive 
information would creep into the autobiography. And the more convincing the CC was as a 
''conversationalist ’, the more dangerous this would be. Overly human-seeming responses on 
the part of the CC could ’ seduce" people to say autobiographical (or other) things that they 
would really prefer to keep private. 

If that happened, and the CC shared the secret with other CCs, and so ultimately with 
inappropriate HBs, should we just shrug and say of the overly-confiding HB: "Idiot! She 
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should have known better!"? Question: if you were to sit down and write your autobiography, 
aren’t there things that you’d leave out? — and not just because they’re boring! 


HBs’ secrets include not only highly sensitive stuff but also gossip. Often, this is to be shared 
only with some people — therefore, to be made available by one’s own CC only to some other 
CCs. Gossip — not necessarily malicious — is a human need, which takes up a huge amount 
of our time. That’s largely why solitary confinement is such a nasty punishment. CCs might 
well be programmed to engage in gossip and eventually ’need ’ it, in the sense that hours of 
conversation without any gossip would lead them to introduce some. But would they enjoy it 
as we do? Would they lead the HB user to imagine that they enjoyed it? If not, then the HB 
concerned wouldn’t enjoy it him/herself as much as usual. 

CCs are recommended, in particular, for lonely (including elderly) people. But would we be 
short-changing them horribly by offering them CC "confidants"? Conversationalists, OK. But 
confidants ? 


In general, there are five questions we should ask: 


1. Could a CC really do or feel X? (Philosophical.) 

2. Could a CC be made to appear to do or feel X? (Largely technological, but needs 
psychological theory too.) 

3. Would the HB user believe the CC could do/feel X? (Socio-psychological.) 

4. Would we want the HB to believe this? (Moral/pragmatic.) 

5. If they did, what effect would this have on their relations with other HBs? (Socio- 
psychological.) 
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Robots Should Be Slaves - Joanna J. Bryson 


Abstract 

Robots should not be described as persons, nor given legal nor moral responsibility for their 
actions. Robots are fully owned by us. We detennine their goals and behavior, either directly 
or indirectly through specifying their intelligence or how their intelligence is acquired. In 
humanising them, we not only further dehumanise real people, but also encourage poor 
human decision making in the allocation of resources and responsibility. This is true at both 
the individual and the institutional level. This chapter describes both causes and 
consequences of these errors, including consequences already present in society. I make 
specific proposals for best incorporating robots into our society. The potential of robotics 
should be understood as the potential to extend our own abilities and to address our own 
goals. 


In this chapter I focus on the ethics of building and using non-human artificial Companions. 
The primary topic of this book is digital Companions, not conventional robots, but both 
pragmatically and ethically the issues are the same. A robot is any artificial entity situated in 
the real world that transforms perception into action. If a digital assistant listens and talks to a 
human, it is a robot — it is an agent, an actor, living in and changing the world. My thesis is 
that robots should be built, marketed and considered legally as slaves, not Companion peers. 
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Digital agents not only change the world by affecting the people they converse with. They 
may also communicate what they learn to others — directly or indirectly through shared 
databases or others' agents. Agents transmit, create and may even destroy information, 
including human opinions and reputations. Digital agents may use the Internet to actively 
purchase goods or services, thus causing the movement of physical objects as well as ideas. 
Finally, some Companion agents really are conventional metal robots with legs and wheels. 
Such robots can do all the things a digital robot can do, and also produce direct physical 
impact on the world — from holding hands or washing windows to breaking dishes and 
falling down stairs. One aspect of direct physical impact is an increased sense of presence for 
the humans. We are, after all, animals with evolved facilities for perception that automatically 
provide assessment and rewards for social actions. 


The question I focus on in this chapter is, what is the correct metaphor we should use when 
thinking about our relationship with robot Companions? By turns in this article I will use two 
different definitions of correct. First, there is the question of what is accurate — what are we 
really producing? And second, what is appropriate — what should we try to build, and how 
should we sell it? 


Robot-oriented ethics are fundamentally different from ethics involving other intelligent 
entities, because they are by definition artifacts of our own culture and intelligence. Perhaps 
unfortunately, we actually have almost as much control over other species and sometimes 
peoples as we do over robots. We as a culture do regularly decide how much and many 
resources (including space and time) we willingly allocate to others. But in the case of 
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biological species there are exquisitely complicated and unique minds and cultures. If these 
minds and cultures are eliminated, they would be impossible to fully replicate. In the case of 
robots, the minds are not there yet, and the culture they would affect if we choose to allow 
them to will be our own. 


Why slaves? 

Slaves are normally defined to be people you own. In recent centuries, due to the African 
trade, slavery came to be associated with racism and also with endemic cruelty. In the past 
though (and in some places still today) slaves were often members of the same race or even 
nation that had simply lost private status. This happened generally as an outcome of war, but 
sometimes as an outcome of poverty. Excesses of cruelty are greatest when actors are able to 
dehumanise those in their power, and thus remove their own empathy for their subordinates. 
Such behavior can be seen even within a small contemporary community of citizen peers, 
when a person in power considers their social standing as an indication of a specialness not 
shared with subordinates. Our culture has for good reason become extremely defensive 
against actions and the beliefs associated with such dehumanisation. 


But surely dehumanisation is only wrong when it's applied to something that really is human? 
Given the very obviously human beings that have been labelled inhuman in the global 
culture's recent past, many seem to have grown wary of applying the label at all. For 
example, Dennett (1987) argues that we should allocate the rights of agency to anything that 
appears to be best reasoned about as acting in an intentional manner. Because the costs of 
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making a mistake and trivializing a sentient being are too great, Dennett says we are safer to 
err on the side of caution. 


Dennett's position is certainly easy to sympathize with, and not only because such generosity 
is almost definitionally nice. As I discuss below, there are many reasons people want to be 
able to build robots that they owe ethical obligation to. But the position overlooks the fact 
that there are also costs associated with allocating agency this way. I describe these costs 
below as well. 

But first, returning to the question of definition — when I say "Robots should be slaves", I by 
no means mean "Robots should be people you own.” What I mean to say is "Robots should 
be servants you own." 

There are several fundamental claims in this paper: 

1 .Having servants is good and useful, provided no one is dehumanised. 

2.A robot can be a servant without being a person. 

3.It is right and natural for people to own robots. 

4.It would be wrong to let people think that their robots are persons. 


A correlated claim to the final one above is that it would also be wrong to build robots we 
owe personhood to. I will not discuss that point at length here; I have at least attempted to 
make that case before (Bryson, 2000). But this corollary follows naturally from my final 
claim above, so I will return to it briefly towards the conclusion of this chapter. 
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Why we get the metaphor wrong 


There is in fact no question about whether we own robots. We design, manufacture, own and 
operate robots. They are entirely our responsibility. We detennine their goals and behavior, 
either directly or indirectly through specifying their intelligence, or even more indirectly by 
specifying how they acquire their own intelligence. But at the end of every indirection lies the 
fact that there would be no robots on this planet if it weren’t for deliberate human decisions to 
create them. 


The principal question is whether robots should be considered strictly as servants: As objects 
subordinate to our own goals that are built with the intention of improving our lives. Others 
in this volume argue that artificial Companions should play roles more often reserved for a 
friend or peer. My argument is this: given the inevitability of our ownership of robots, 
neglecting that they are essentially in our service would be unhealthy and inefficient. More 
importantly, it invites inappropriate decisions such as misassignations of responsibility or 
misappropriations of resources. 

Many researchers want to build AI that would have moral agency — that is, to which we 
would owe ethical obligations as we do to a person. Helmreich (1997) suggests this desire to 
create Artificial Life is most present in researchers that are middle-aged males with a fixation 
on an individual capacity for creating life. While there seems to be a certain empirical 
plausibility of Helmreich's hypothesis, in my own experience many people who are not 
themselves roboticists or middle-aged still seem to have a shockingly strong tendency to 
believe they owe ethical obligation to robots. 
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I was astonished during my own experience of working on a (completely non-functional) 
humanoid robot in the mid 1990s, by how many well-educated colleagues volunteered, 
without prompting and immediately on seeing or even hearing about the robot that 
unplugging such a robot, would be unethical. Less anecdotally, popular culture contains 
many examples of heroic, conscious robots examining the worth of their own lives. For 
example, in the original Star Wars (A New Hope), there is a running theme concerning the 
slavery and even torture of robots which is at times explicitly voiced by the 'droid character 
C3P0 1 This theme is not resolved in that movie, nor is it so prominent in any of the Star 
Wars sequels. However in Blade Runner, The Bicentennial Man, A.I.: Artificial Intelligence, 
and several episodes of Star Trek: The Next Generation featuring the robot crew member 
Data, the central question is what it takes for a robot to be a moral agent. Traditional literary 
criticism of science fiction holds that the artistic examination of alien or artificial sentience is 
a mechanism for examining by proxy humanity and the human condition. However, many 
producers and consumers of science fiction consider themselves futurists, examining future 
rather than (or along with) present moral dilemmas. Whatever the artistic intention, from my 
experience as a roboticist, it seems that a large proportion of science fiction consumers are 
comfortable with a conclusion that anything that perceives, communicates and remembers is 
owed ethical obligation. In fact, many people seem not only comfortable, but proud of having 
come to this realization, and willing to defend their perspective angrily. 


Bryson and Kime (1998) have argued that both deep ethical concern for robots, and 
unreasonable fear of them, results from uncertainty about human identity. Our identity 
confusion results in somewhat arbitrary assignments of empathy. For example, contemporary 
cultures — including some scientific ones — often consider language both necessary and 
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sufficient for human-like intelligence. Further, the term conscious (by which we mostly seem 
to mean "mental state accessible to verbal report") is heavily confounded with the term soul, 
(meaning roughly "the aspect of an entity deserving ethical concern”). Thus, for example, 
animal 'rights' debates often focus on whether animals are conscious, with the explicit 
assumption that consciousness automatically implies ethical obligation (Rollin, 1998). Of 
course, neither the term consciousness nor soul is precisely or universally defined. Rather, 
these tenns label concepts formed by accretion over millenia as we have attempted to reason 
about, analyse and describe ourselves. 


I would see Helmreich's diagnosis then as just part of a much wider problem. It is not only 
that a few prominent middle-aged male artificial life researchers may have fantasies of 
autonomously creating human-like life. Entire cultures — including perhaps the global one — 
could be expected to take pride in such an achievement, as they did in the conquest of space 
or in colonialism and empire. But this desire is a symptom of a larger confusion about 
whether AI could or should be a surrogate for human life and human responsibility. 


Since this book examines artificial Companions, in the rest of this chapter I concentrate 
primarily on the best role for robots in the household — the costs and benefits of different 
attitudes towards robots at the individual and commercial level. 
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Costs and benefits of mis-identification with AI 


The individual level 

Over-identification with AI leads to a large range of category errors which can significantly 
bias decision making. This is true at decision-making levels ranging from the individual to 
the national and super-national. Already we have seen commercial willingness to exploit 
human empathy for AI objects such as Tamagotchi 'pets'. Like every other potential 
distraction — from radios to children — Tamagotchi have led to fatalities from automobile 
accidents. But there are more important policy considerations than the occasional sensational 
headline. 


At the personal level, the cost of over-identification with AI should be measured in: 

1 .the absolute amount of time and other resources an individual will allocate to a virtual 
Companion, 

2. What other endeavors that individual sacrifices to make that allocation, and 

3. Whether the tradeoff in benefits the individual derives from their engagement with the AI 

outweigh the costs or benefits to both that individual and anyone else who might have 
been affected by the neglected alternative endeavors. 


This final point about the tradeoff is somewhat convoluted, so I want to clarify why I make so 
many qualifications. My arguments in this chapter derive primarily from the default liberal- 
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progressive belief that the time and attention of any human being is a precious commodity 
that should not be wasted on something of no consequence. However, life is more 
complicated than simple principles. Clearly some people are socially isolated, and it has long 
been demonstrated that isolated elderly people really are healthier with empathic, social non¬ 
human Companions (Siegel, 1990fi Also well in evidence is that some people sometimes 
engage in harmful, antisocial behavior. There is at least preliminary evidence that increased 
Internet access directly correlates to the recent impressive decreases in the levels of sexual 
assault (D’Amato, 2006). Thus it is at least possible that in some cases the 'cost' of over- 
identification with AI could in fact be negative. In other words, becoming overly emotionally 
engaged with a robot may in some cases be beneficial, both for the individual and society. 


But returning to the default liberal-progressive view, my concern is that humans have only a 
finite amount of time and drive for forming social relationships (Dunbar, 1997), and that, 
increasingly, we find ways to satiate this drive with non-productive faux-social 
entertainment. Putnam (2000) documents the massive decline in what he calls "social 
capital" in the Twentieth Century. Although Putnam originally ascribed this decline to 
increasingly dynamic societies and a lack of'bridging' individuals between communities, his 
own further research failed to sustain this theory. A simpler explanation seems likely: that the 
drive to socialize is increasingly being expended on lower-risk, faux-social activities such as 
radio, television and interactive computer games. Each of these technologies has 
progressively increased the similarity between individual entertainment and true interpersonal 
interactions. 
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In fact, more recently online social networking has actually reintroduced a human element to 
technological entertainment. Facilitating human-human interactions could be a component of 
a domestic robot's behavioral repertoire, too. However, this complication is irrelevant to the 
main argument of the present chapter. I am considering here autonomous robots — actors in 
their own right. Virtual or physical avatars being teleoperated by their owners' friends or 
family are already slaves in such an extreme sense that I don’t believe anyone would argue 
they deserve moral agency. Such robots, called avatars performs precisely the actions 
detennined for them by the people they represent, and have no internal motivation 
whatsoever. 


The institutional level 

The individual-level social cost of mis-identification with robots is the economic and human 
consequence of time, money and possibly other finite resources being given to a robot that 
would otherwise be spent directly on humans and human interaction. The cultural- or 
national-level costs obviously include the combined individual costs (or benefits) of the 
citizens of that culture or nation. But it is possible that in addition similar errors might also 
arise at a higher institutional level. I am not particularly concerned that the enthusiasm for the 
creation of life, or of super-human, super-obedient robot citizens, might actually attract 
government research funding or legislative support. However, both the USA and the 
European Union have openly made research on autonomous cognitive systems a high funding 
priority. In the US this has been accompanied by both funding and media attention on making 
certain the systems (in the US, mostly battlefield robotics) are capable of moral decision 
making. Government-funded robotics professors are openly suggesting that robots might 
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make more ethical decisions than humans in some battlefield situations (Wallach and Allen, 


2008; Fong et ah, 2008). 


While I have no problem with the use of Artificial Intelligence to complement and improve 
human decision making, suggesting that the AI itself makes the decision is a problem. Legal 
and moral responsibility for a robot's actions should be no different than they are for any 
other AI system, and these are the same as for any other tool. Ordinarily, damage caused by a 
tool is the fault of an operator, and benefit from it is to the operator's credit. If the system 
malfunctions are due to poor manufacturing, then the fault may lie with the company that 
built it, and the operator can sue to resolve this. But creating a legal or even public-relations 
framework in which a robot can be blamed is like blaming the private soldiers at Abu Ghraib 
for being "bad apples". Yes, some apples are worse than others and perhaps culpably so, but 
ultimate responsibility lies within the command chain that created the environment those 
privates operated in. Where the subject is machines we build and own, then the responsible 
role of the organisation is even clearer. We should never be talking about machines taking 
ethical decisions, but rather machines operated correctly within the limits we set for them. 


I can see no technological reason why people should expect moral agency from an 
autonomous mobile gun when they do not expect it from automatic tellers (banking 
machines) or conventional automatic dishwashers. Of course shooting is more dangerous than 
cleaning, but that doesn’t make guns more moral agents than sponges. As we increase the on¬ 
board sensing, action and logic in these tools, can there really come a point where we may 
ourselves abrogate ethical responsibility for directing the taking of lives? In my opinion, no. 
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Advanced weapon systems may seem exceptional, but I expect these technologies and issues 
will rapidly move into the civilian sector. This was the history of air flight: Originally the 
domain of engineers and hobbyists, once flight was conquered it was immediately applied to 
warfare. Following World War I the technology was quickly exploited by the creation of an 
industry for both business and leisure services. If we allow robots to bear their own 
responsibility for their behavior in foreign battlefields, we will soon face the same issues at 
home as we find them working in police forces and post offices. 


We do not have to wait for the presence of advanced AI to see the consequences of 
responsibility passing away from humans. Consider existing automated (or at least unstaffed) 
railway stations. Ordinarily they work well, perhaps providing ticket services in more 
languages than a small rural station might otherwise provide. But when a train fails to make a 
scheduled stop at six on a Sunday morning, there is no one to apologise and provide a 
replacement taxi service, making sure you get to the airport for your flight. 


At the national and institutional level, this is the moral hazard of being too generous with 
personhood. Not only does automation save in staff cost, but also it reduces corporate 
responsibility for service from the level of the reasonable capacity of a human to that of a 
machine. Further, the capacities of a machine are largely determined by those who choose 
how much they will pay for it. At the personal level, the moral hazard is choosing less rich 
interactions with lower social impact because robotic interactions are more predictable and 
less risky. If you do something stupid in front of a robot, you can delete the event from its 
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memory. If you are tired of a robot or just want to go to bed early, you don’t need to ask its 
opinion, you can just turn it off. 


While there may well be people and institutions for whom such automated services are the 
only available option, the question is how many will be tempted unnecessarily into being less 
responsible and productive members of society. As with the level of government welfare 
benefits, there probably is no ideal target level of faux-humanism we can provide in a robot 
that both helps everyone who really needs assistance, while not tempting anyone away from 
social contribution who does not. Finding the appropriate level then is difficult, but my claim 
is that at least in the case of robotics, it can be located more easily by providing accurate 
public infonnation. 


Getting the metaphor right 

Why, if robots are so hazardous should we want to include them in our lives? Because 
servants have the potential to be useful. When there was greater income disparity in the UK 
and domestic chores such as cooking were far more time consuming, human servants were 
much more common. Laslett (1969) found that approximately 30% of all households had 
servants in a set of British villages surveyed from 1574-1821. At that time, tasks such as 
cooking and cleaning for even a moderately-sized family could take two people 12 hours a 
day every day (Blair, 2008). Where wives and other kin were not available to devote their full 
time to these tasks, outside employees were essential. 
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Contemporary cooking and cleaning takes far less time than in the days before electricity, gas 
or running water. Nevertheless, many aspects of food preparation are still often outsourced. 
Now this work is done in factories, by agricultural labour or in restaurants rather than by in- 
home domestic servants. The European Commission has chosen to invest heavily in cognitive 
systems for domestic robotics, partly due to anticipated foreign demand and perceived local 
expertise, but also in an effort to increase the productivity of its own workforce. 


Anyone who has worked recently with robots knows that they are unlikely to be deployed 
dusting plants or cleaning fine china in the near future. The most likely initial introduction of 
domestic robotics will be partial or total replacement of expensive human help in tasks that 
require less real-time planning and dexterity. Examples are physical support for the infirm, 
minders and tutors for children, and personal assistants for those with challenged working 
memory capacity, whether that challenge is due to disease or to distraction. Note that none of 
these applications requires anthropoid robots. In fact, a personal assistant might take the form 
of a personality-based video interface for a smart home. Avatars for the AI might flicker to 
the nearest fixed screen location when called; AI may monitor embedded household sensors 
for danger, injury, or just a forgotten lunch. In homes with more than one occupant, a variety 
of robot personalities can use the same infrastructure each serving their own user. 


In my opinion, communicating the model of robot-as-slave is the best way both to get full 
utility from these devices and to avoid the moral hazards mentioned in the previous sections. 
Of course some people will still talk to their robots — some people talk to their plants and 
others their door knobs. But those people have neighbours and relatives who know that the 
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plants and door knobs don’t understand. These people help their overly-conversant relative or 
friend keep that fact somewhere in their minds. Similarly, our task is not to stop people from 
naming or petting their robots. Our task is to ensure that the vast majority of the population 
understands that robots are just machines, and that one should spend money and time on them 
as is appropriate to their utility, but not much more. 


Of course, some people may for a variety of personal, historical or cultural reasons object to 
having any form of servant or slave in their home. In that case, there is another metaphor that 
might be useful — that of the extended mind (Clark and Chalmers, 1998). If the robot has no 
goals except for those it assumes from you, then there are rational arguments to be made that 
robot is just an extension of yourself. I have previously made an analogous argument that the 
Semantic Web should be viewed neither as a giant database nor as a set of agents to be 
knitted together in a complex democratic community in order to provide a simple service. 
Rather, the Semantic Web should be thought of as snippets of intelligence that can be used to 
augment the capabilities of a user's own personal assistant (Bryson et al., 2003). Only the 
personal assistant has a motivational structure, and this it inherits from the goals of its user. 


It is a fairly simple extension of this argument to say that the digital personal assistant itself 
could be considered an extension of the user. Our goals, beliefs, perception and capacity for 
action can all be extended or made more reliable through a range of robotic servants — or if 
one prefers, services. In extending what we do and know, robots could also be seen to extend 
who we are. 
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Don't we owe robots anything? 


If servants are such a great idea, shouldn’t we just hire more human ones? To some extent, we 
already do. Our economy has a large service component. Many tasks that a hundred years ago 
were performed by live-in servants are now largely performed by strangers outside the home, 
such as food preparation and clothes manufacture. Other tasks are already performed at least 
in a large part by machines, such as washing laundry, mowing lawn, or keeping an 
appointment calendar. 


The poor are richer now than they once were. Virginia Woolf paid her live-in servants only 
1% of her own annual income of £4,000 (Blair, 2008). Even a part-time domestic servant 
who was willing to take 1% of a modem professional salary would be unlikely to do many 
things that could not better be done with machines or outside services. 


But the most difficult thing with human servants is of course the fact that they really are 
humans, with their own goals, desires, interests and expectations which they deserve to be 
able to pursue. Humans living and working together but set as not each other's equals are 
often vulnerable to frustration and exploitation. 


But what about the robots? Would they not feel frustrated? Would they not be exploited and 
abused? 
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Remember, robots are wholly owned and designed by us. We determine their goals and 
desires. A robot cannot be frustrated unless it is given goals that cannot be met, and it cannot 
mind being frustrated unless we program it to perceive frustration as distressing, rather than 
as an indication of a planning puzzle. A robot can be abused just as a car, piano or couch can 
be abused — it can be damaged in a wasteful way. But again, there's no particular reason it 
should be programmed to mind such treatment. It might be sensible to program robots to 
detect and report such ill treatment, and possibly to even avoid its abuser until its owner has 
been notified (assuming the abuser is not the owner). But there is no reason to make a robot 
experience suffering as part of the program to generate such behavior. 


Owners should not have ethical obligations to robots that are their sole property beyond those 
that society defines as common sense and decency, and would apply to any artifact. We do 
not particularly approve of people destroying rare, fine cars with sledge hammers, but there is 
no law against such behavior. If a robot also happened to be a particularly fine piece of art 
then we would owe it the same obligations we owe other pieces of art. 


Robot owners should not have obligations, but ensuring this is the responsibility of robot 
builders. Robot builders are ethically obliged — obliged to make robots that robot owners 
have no ethical obligations to (Bryson, 2000). A robot's brain should be backed up 
continuously off-site by wireless network; its body should be mass produced and easily 
interchangeable. No one should ever need to hesitate an instant in deciding whether to save a 
human or a robot from a burning building. The robot should be utterly replaceable. Further, 
robot owners should know their robots do not suffer, and will never 'die' even if the rest of 
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their owner's possessions are destroyed. The robot's brain state should be preserved off site. 
The robot can return to function as before as soon as a new body can be acquired, though it 
may need some retraining if there is a new domicile to inhabit or slight variations between 
bodies. Robots then can be a relied on as no more than extensions of their owners. They 
should not be anthropoid if that can be helped, and their owners should have access to the 
robots' program-level interface as well as its more socially-oriented one. This will help the 
owners form a more accurate, less human model for reasoning about their assistive agents. 


We do then have obligations regarding robots, but not really to them. Robots are tools, and 
like any other artifact when it comes to the domain of ethics. We can use these tools to extend 
our abilities and increase our efficiency in a way analogous to the way that a large proportion 
of professional society historically used to extend their own abilities, but with fewer ethical 
and logistical hazards. Hopefully, we can continually increase the number of service owners 
in our society, so a smaller proportion of everyone’s time can be spent on mundane or 
repetitive tasks if they do not enjoy them. In that case, a larger proportion of time and 
resources can be spent on useful processes, including socializing with our colleagues, family 
and neighbours. 


Conclusions 

Why do people want robots to be their peers? Is it perhaps because they want a 'peer' that 
will never argue, or at least never be smug when it wins? A fairy god-parent smarter than 
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themselves, that they can nevertheless ultimately boss around and pen up like a pet dog? If 
so, such narcissism is probably mostly hannless, and perhaps a good thing for the dogs. But 
in a liberal democracy we tend to think of every citizen’s life and mind as a valuable resource. 
Wasting that resource 'socializing' with artifacts would be a great loss. 


Robots should rather be viewed as tools we use to extend our own abilities and accelerate 
progress on our own goals. An autonomous robot definitionally incorporates its own internal 
motivational structure and decision mechanisms, but we choose those motivations and design 
the decision-making system. All their goals are derived from us. 


I have argued here that robots are often overly personified. First, this is because of our desire 
to have the power of creating life. Second, this is because we are not certain what it means to 
be human, so we currently offer the term to anything that senses, acts, remembers and speaks. 
Given the errors in dehumanisation that have been broadly made in the very recent past — in 
fact, sometimes in the present — the desire to avoid such mistakes is laudable. Yet ironically, 
extending the title human to something which is not only serves to further devalue real 
humanity. 


The objective of this chapter has been to persuade roboticists and robotophiles — now, while 
this industry is in its early stages — that calling a robot a moral agent is not only false but an 
abrogation of our own responsibility. I have also demonstrated that these problems are 
already in our society: in the current research funding for ethical battlefield robots, and in the 
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commercial exploitation of human empathy for artificial characters. My conclusion is that we 
are obliged not to the robots, but to our society. We are obliged to educate consumers and 
producers alike to their real obligations with respect to robotics. 
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Wanting the impossible: the dilemma at the heart of intimate human-robot 
relationships - Dylan Evans 

Abstract 

In a recent book entitled Love and Sex with Robots, the British scholar David Levy has 
argued that relationships with robot Companions might be more satisfying than relationships 
with humans, a claim which I call "the greater satisfaction thesis" (GST). The main reason 
Levy provides in support of GST is that people will be able to specify the features of robot 
Companions precisely in accordance with their wishes (which I call the total specification 
argument or TSA). In this paper, I argue that TSA is wrong. In particular, the argument 
breaks down when we consider certain behavioral characteristics that we desire in our 
partners. I illustrate my argument with a thought-experiment involving two kinds of robot - 
the FREEBOT, which is capable of rejecting its owner permanently, and the RELIABOT, 
which is not. 


Introduction 


The idea of romantic relationships between humans and robots has long been a province of 
science fiction. In films like Blade Runner (1982) and Bicentennial Man (1999), for 
example, mutual love between a human and a robot is portrayed as both possible and 
beautiful. 29 Outside the realm of literary and film criticism, however, there was until recently 
very little serious discussion of the technological, social, legal and ethical questions that 
robots capable of such relationships might pose. The publication of Love and Sex with 
Robots has gone some way to changing this (Levy, 2007a). While the topic is not yet in the 
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main stream of debate among roboticists, Levy's book has already inspired at least one 
international scientific conference 30 and attracted comment by researchers in a range of 
scientific fields. 


At several points in his book, Levy suggests that romantic relationships with robot 
Companions will be more satisfying than romantic relationships with humans. I will refer to 
this claim as the greater satisfaction thesis (GST): 


GST: It is likely that romantic relationships with robot Companions will be more satisfying 
than romantic relationships with human Companions. 


Levy does not specify the scope of GST very precisely, but it is clear that he means it to refer 
to more than just a few exceptional people. Some of those interviewed for a British 
documentary about people who appear to have romantic relationships with sex dolls seem, on 
the face of it, to prefer these dolls to human partners (Holt, 2007). If a documentary maker 
can find some people who prefer inanimate sex dolls to human partners, then it seems very 
likely that, once the robot Companions envisaged by Levy become available, there will be a 
few people at least who prefer romantic relationships with these robots over romantic 
relationships with humans. If this was all that Levy meant, it would not be very surprising. 
But the fact that Levy clearly thinks his claim is a bold one implies that he intends the scope 
of GST to be quite wide. It is plausible to interpret Levy as claiming, then, that many people 
will find romantic relationships with robots more satisfying than romantic relationships with 
human beings.The main reason Levy provides in support of GST is that people will be able to 
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specify the features of robot Companions precisely in accordance with their wishes. So, for 
example, people who are worried about infidelity will be able to request robot Companions 
programmed to be completely faithful. Alternatively, those who prefer the frisson of 
uncertainty may specify that their robot Companion be programmed to have some probability 
of being unfaithful. I will refer to this as the total specification argument (TSA): 


TSA: One reason for supposing GST to be true is that we will be able to specify the features 
of robot Companions entirely in accordance with our wishes. 


In this paper, I will argue that TSA is wrong. Robot Companions, at least those of the kind 
that Levy envisages, will not be more satisfying than relationships with humans. Before I 
proceed, however, I must first make it clear that the grounds for my objection have nothing to 
do with empirical facts about robot technology. The reason that robot relationships will not 
be more satisfying than relationships with humans has nothing to do with any technical 
limitations inherent in robots. There may or may not be such technical limitations, but I tend 
to agree with Levy when he dismisses objections of this kind, since we do not know enough 
about the limits of technology in general to be able to rule out, with any degree of confidence, 
any of the particular technical features that he discusses in his book. My objection to Levy's 
claim about the potentially greater satisfaction that we may obtain from relationships with 
robots is based, not on claims about robot technology, but on claims about the what people 
tend to want from an intimate relationship. 
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It is worth spelling out here the kind of robots that Levy has in mind. For throughout his 
book, Levy only discusses a relatively narrow class of possible robot Companions. In 
particular, the robot Companions that Levy envisages are all: 


• humanoid 

• made in a factory according to specifications provided by the user 

• consumer products designed to serve the needs and wishes of humans 

• private property 


Thus Levy does not consider, for example, robots that build themselves, or robots that exist 
for their own purposes, or robots that have any kind of right to leave their owner's service. 
Such characteristics may be too advanced to merit discussion in what Levy claims is an 
analysis of technology that is possible within a few decades, but the uncertainty surrounding 
the speed of technical developments is too great to make any firm pronouncements here. 
Besides, some of the other features that Levy thinks will be feasible within that time span, 
such as learning the preferences of a user by observing their behavior, might be just as 
complex as the ability to build copies of itself. So the fact that Levy restricts himself to 
considering just one kind of robot Companion, in just one of the various possible commercial 
and legal frameworks in which robot Companions might operate, can at times make the book 
seem more like one long advert for a product that doesn’t exist yet. Nevertheless, since my 
aim in this paper is to criticise Levy's claim, I too will restrict myself to a discussion of the 
type of robot discussed in his book. 
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This paper is structured as follows. In the next section, I discuss what I mean by “technical 
objections” to GST in order to make it clear, by contrast, the kind of objection to GST that I 
advance in this paper. In the following sections I outline my own objection to GST, and 
illustrate it by proposing a thought-experiment involving a choice between two different 
kinds of partner robot - one which is capable of permanently rejecting its owner (a 
FREEBOT) and one which is not (a RELIABOT). 


Technical objections 


By “technical objections”, I mean objections to Levy's claims that are based on how difficult 
it would be to build robots with a particular feature. Take Levy's discussion of robot fidelity, 
for example. A robot that was guaranteed to be completely faithful to its owner would 
require some means of identifying its owner with 100% accuracy. We might argue that such 
a capacity is unlikely on various empirical grounds: 


i. No clear path from current technology: One set of arguments is based on extrapolations 
from specific technologies that are currently in existence. For example, at least one current 
facial recognition system has been shown to be 100% accurate when tested with a set of still 
photographs taken from the Internet (Jenkins and Burton, 2008). However, when it comes to 
recognising faces from a moving video camera in real time under a wide range of lighting 
conditions (all conditions that would apply to face recognition in a humanoid robot), accuracy 
rates fall dramatically, and fall far short of the recognition capacities of humans. It could be 
argued that there is currently no clear path from the relatively poor accuracy rates of facial 
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recognition possessed by mobile robots to the 100% accuracy rates that they would have to 
enjoy for robots to be capable of perfect fidelity. 


ii. General engineering principles : A second set of arguments might start, not with specific 
technologies, but with general engineering principles. For example, it could be argued that 
no technology can ever be 100% accurate because there are always multiple failure points in 
any machine. 


iii .Hacking and deception: A third set of arguments could appeal to the possibility that even 
if recognition systems could be built that were, in fact, 100% accurate under most 
circumstances, they would always be vulnerable to hacking and active deception. Let us 
imagine, for example, that a robot recognises its owner by means of some biometric system 
such as iris-recognition, or by some other means such as detecting an RFID tag with a unique 
user ID (perhaps embedded subcutaneously). Let us suppose, further, that this system never 
gave a false positive. Now, it would always be possible to fool such a robot by various 
means. For example, a malicious user could hack into the file which the robot used to 
associate the category ’owner ’ and the relevant iris scan or user ID. Alternatively, they 
could produce a fake retina that mimicked the pattern of the owner's iris, or copy the owner's 
RFID tag. 


All these arguments have some merit. But to take these as the only arguments against Levy's 
claim that robots will be potentially more satisfying than human Companions is to agree with 
Levy that, if such technical hurdles are overcome, robots may indeed prove more satisfactory 
Companions than humans. Since it is this assumption that I wish to challenge, I will suppose 
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for the sake of argument that all the technical limitations can be overcome. Let us suppose, 
for example, that it is possible to design a robot capable of 100% accuracy in recognising its 
owner, and therefore capable of being completely faithful if required. Would this give us 
good grounds for supposing that relationships with robots will be more satisfying than 
relationships with humans? I think not. The reason, as I explain in the following section, has 
to do with certain fundamental features of human emotions. 


Wanting to be wanted 


Intimate relationships between humans are difficult and complex things, in which conflict 
and pain are pervasive features. If any evidence were required to support this claim, it could 
be found in abundance in the advice columns of agony aunts, in the statistics for divorce, and 
in the long tradition of tragic love stories in plays, novels and more recently, film. Even in 
the happiest and most long-lasting marriages, spouses acknowledge the persistence of stress 
and conflict (Ammons & Stinnett, 1980). 


GST suggests that some of the unpleasant features of human-human relationships will be 
absent from, or at least diminished in, human-robot relationships, providing the robots are 
appropriately designed. This implies that the problem lies in some contingent feature(s) of 
human Companions that prevents them from fully satisfying our desires, and that appealing 
robots can be designed without this feature or set of features. According to TSA, the 
contingent features of human Companions are their failure to fully match our preferences. 
Presumably, the line of reasoning is as follows: 
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People do not come made to order, and so we must settle for some compromise 
between what we really want and what we can get. This difference between what 
we really want and what we have to settle for is at least partly responsible for the 
less pleasant aspects of human relationships. Robots, on the other hand, are 
made to order. Unlike people, they can be designed to match our preferences 
perfectly. Therefore, those problematic aspects of human-human relationships 
that are caused by the gap between desire and reality will be absent from human- 
robot relationships. 


This argument may have some degree of plausibility when we consider only the physical 
characteristics that we desire in our partner. Someone may want a boyfriend who is very tall 
and muscular, or a girlfriend with large breasts, but may not be able to find someone with all 
the physical characteristics they desire (or they may find someone with these characteristics, 
but that person does not reciprocate the affection). In this case, it would presumably be easy 
to specify a robot partner with all the right proportions. However, the argument breaks down 
when we consider certain behavioral characteristics. For among the various desires that most 
people have regarding relationships are certain second-order desires (desires about desires), 
including the desire that one's partner has freely chosen to be one's partner, and has not been 
coerced into this decision. 


With the kind of robots that Levy envisages - robots that are ordered from a factory, and 
purchased as consumer goods - it is clear that this desire, at least as stated above, cannot be 
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fulfilled. These robots do not choose their owners, at least initially. Nevertheless, if the 
robot had the capacity to reject its owner, robots that did not exercise this capacity might be 
said to choose their owners in some ongoing sense, even if there was no choice initially. In 
the absence of an initial choice on the robot's part, therefore, the robot's capacity to reject its 
owner would be crucial for satisfying the owner's desire that he or she has been freely chosen 
(if the owner does have this desire). 


Levy anticipates this idea when he states that an important part of loving relationships is "the 
possibility of failure or denial." Indeed. It does seem that the feeling of being loved 
romantically depends crucially on the belief that this love is not unconditional, i.e. that it is 
contingent on the way you behave, and might disappear. This may seem counter-intuitive; 
after all, people often say they want unconditional love from their partner, or say they love 
their partners unconditionally. Nevertheless, if one's partner really did love unconditionally, 
and would put up with any treatment, no matter how badly one behaved, I suspect that the 
charm would rapidly fade. This point is crucial to my argument in this paper, so let me 
elaborate on it. 


People often say they want their partners to be reliable, faithful, always there for them, never 
to leave them, and so on. But they want these qualities to be the fruit of an active and 
ongoing choice. The most effective way to signal that there is a real choice involved here is 
for the partner to drop hints that there is a genuine possibility that they could leave, if they 
ever wanted to. So, paradoxically, for people to feel secure that their partners freely choose 
to be with them, and not with anyone else, they must occasionally be made aware of the 
partner's freedom by occasional rejections (huffs, moods, and so on), and by the occasional 
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sign that one's partner finds other people attractive too. It can be very painful when one's 
partner is grumpy, or seems attracted to someone else, but it is also strangely compelling. 
“Treat them mean to keep them keen”, as the saying goes. 


The paradoxical dynamics that this complex attitude can engender may play some role in 
explaining the mixed blessings of marriage. Traditionally, marriage represented the 
renunciation of any further choice. This promise has perhaps been honoured more in the 
breach than in the observance, but it appears that many people continue to believe that their 
partner is more fully ‘theirs ’ when married. If the desire for fidelity in one's partner were a 
simple matter, the renunciation of choice signalled by marriage would only serve to make 
people happier. If, however, the perverse dynamics of mate choice just described apply, 
marriage should be a more ambiguous benefit, for the gain in reliability would be partly 
offset by the reduction in the interest that such reliability brings. Which of these two 
alternatives is more in line with the facts? I know of no scientific research on this matter, but 
folk wisdom and popular culture provide some anecdotal support for the latter prediction. 
There is an old saying that “if love is a disease, then marriage is the cure”. 31 In the film The 
Witches of Eastwick, for example, the diabolic figure of Darryl Van Horne points to the same 
idea in more graphic terms when he exclaims: "Don’t believe in marriage. Good for the man, 
bad for the woman. She dies. Then the man runs around complaining that he's fucking a dead 
woman, and he’s the one who killed her!" It is an old story; a man finds a woman attractive 
because she is dynamic and independent, and works hard to win her affection. Eventually, he 
persuades her to marry him. Once they are married however, the dynamism and 
independence that he previously found attractive are now perceived also as a threat, and he 
does his best to domesticate her. If he succeeds, however, he complains that his wife is no 
longer so interesting. This effect is no doubt caused by a whole range of factors, from 
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psychological factors such as habituation to biological mechanisms such as the Coolidge 
effect, but one of the ingredients in this heady mix is probably the sense that there is less 
chance that the person will leave, and therefore less to fight for. 


While these considerations are certainly far from being decisive, they do lend some support 
to the intuition that the sense of being freely chosen by one's partner is an important 
detenninant of the satisfaction that one derives from a relationship. Although people 
typically want commitment and fidelity from their partners, they want these things to be the 
fruit of an ongoing choice, rather than inflexible and unreflexive behavior patterns. An 
obvious objection to Levy's GST, then, is that robots will not be able to provide this crucial 
sense of ongoing choice, and that relationships with robots will therefore be less satisfactory 
than relationships with humans with respect to this feature. Let us call this “the free-will 
objection ” (FWO). 


A tale of two robots 


As already noted, Levy partly anticipates the free-will objection when he states that an 
important part of loving relationships is "the possibility of failure or denial”. His response to 
FWO is that partner robots will need to be able to mimic human "capriciousness." However, 
this is too quick, and it begs a fundamental question - namely, how far should the mimicry 
go? 
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The various answers to this question may be classified in various ways. One way is to divide 
them into two groups, depending on whether or not the mimicry of capriciousness extends to 
the point at which the robot can reject its owner permanently. To make these two alternatives 
more vivid, picture them as two types of partner robot, the RELIABOT and the FREEBOT: 


1. With the RELIABOT there is a point at which the mimicry of capriciousness stops; 
the robot might appear to reject you for a while, but it always relents in the end. 


2. With the FREEBOT the mimicry of capriciousness is not bounded; there is always 
a chance that the robot might reject you permanently. 


To see why Levy's proposed solution to FWO is inadequate, consider the following question: 
if you were going to buy a partner robot, which model would you choose - the RELIABOT 
or the FREEBOT? 


This dilemma has already been anticipated by connoisseurs of robot sexual fantasies. There 
is, for example, a website in which users can specify the characteristics of an imaginary 
fembot that they wish to order. After specifying the ‘type of body' and the ‘ personality ’ 
required, the user is asked to specify the ‘controlprogramming ’ desired by choosing from the 
following options (Anonymous, 2008): 
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1. Total Overriding ( Functions separate from main AI, will force robot to obey to the 
letter a command regardless of will of primary AI. Not recommended as will cause 
inefficiency in Unit.) 

2. Integrated (Unit will have control programing integrated into primary AI. Unit will 
see it's sole purpose is to serve and please you.) 

3. Subliminal (Unit will like you, but will not see it's sole purpose is to serve you, it 
will however obey any suggestion or command you give it, viewing it as “a good 
idea”.) 

4. None (No control program. WARNING: Not recommended. Androtech will not be 
held accountable for the outcome if this is selected 

The choice here differs slightly from the choice I am considering under the RELIABOT 
versus FREEBOT headings, but it hints at a similar dilemma. The first three options 
correspond (roughly) to types of RELIABOT, while the fourth option is a FREEBOT. Users 
of the website are in a similar position to that of the consumer who wishes to purchase one of 
Levy's imaginary partner robots - he or she must choose the level of capriciousness. And 
here lies the rub. For if we choose the RELIABOT, the considerations advanced in the last 
section of this paper suggest that we will soon tire of the robot, and come to take it for 
granted, since it will lose the frisson associated with the contingency of a human Companion. 
On the other hand, if we choose the FREEBOT we spend our money on an unreliable 
product. Indeed, the FREEBOT might be a difficult product to sell. 
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The question of whether the charms of a RELIABOT will rapidly fade is an empirical 
question, and in the absence of good data about relationships with such robots we can only 
speculate. Nevertheless, if the observations above about the tendency of people to tire of 
relationships with people who become too predictable are correct, it is plausible that the same 
will be true of relationships with robots. There is also the possibility that RELIABOTS could 
create favourable conditions for the emergence of cruelty. The urge to vent one's frustration 
on people and objects in the close vicinity is often curtailed by the thought that it may lead 
someone to retaliate, or (which may be worse) to cease interaction. If RELIABOTS are not 
allowed to retaliate (a capacity that would raise all sorts of ethical dilemmas that I will not 
discuss here), there will be no effective sanction open to them for punishing cruelty (since by 
definition they cannot make credible threats to walk away forever). This is not a positive 
incentive to cruelty, but the mere absence of a disincentive may be enough to foster cruelty. 
Again, whether it does in fact have this effect, and how frequently, is an empirical question, 
but it also has moral implications. For example, if the way that people with robot partners 
treat those partners influences the way they treat other people, then those with RELIABOT 
partners might tend to be more cruel (and, more generally, more instrumental) in their 
relationships with other people. It is easy to see how this might happen. First, some behavior 
that is unacceptable when directed at a human being might be considered more acceptable 
when directed at a robot. The similarity of humans and robots might then lead this behavior 
to become more common in human-human relationships. If so, then perhaps the rather 
unsettling cartoons by Itsuki Takashi involving "Amputee Robot Girl Bondage" are an 
ominous sign of things to come (Takashi, 1986). 


If the RELIABOT has all these difficulties, then how does the FREEBOT alternative fare? 
The owner's knowledge that the FREEBOT might reject him or her permanently might 


140 



reduce or eliminate the risks of boredom and cruelty just discussed, but it is not clear whether 
this advantage would be worth the price. Imagine if the FREEBOT came with the following 
warning label: 


Rossum's Universal Robots does not guarantee the ongoing compliance of this 
fembot with your wishes. Bad treatment may cause her to enter permanent 
nonresponsive mode. Resetting her after this will erase all memory of previous 
interactions. 


Would you buy a product that carried such a label? Perhaps you would, if you valued the 
feeling of being actively chosen that the knowledge of your robot's capacity for pennanent 
rejection might lead to. But consider the following scenario. You have invited a friend for 
dinner. The friend arrives at your flat and spots your fembot lying on the couch. “What's 
that?” he asks. “Oh, that's my fembot,” you reply. “Why is she just sitting there?” asks your 
friend. “I treated her badly,” you reply; “she's entered permanent nonresponsive mode now. ” 
Your friend pauses for moment, and then asks: “How much did you pay for her?” You look a 
bit sheepish. “About ten thousand bucks, ” you murmur. Your friend raises his eyebrows. 
“Wow, that's a lot of money for a useless hu nk of metal!” Once again, it is an empirical 
question as to how many people would buy a product for which the lack of a guarantee was 
supposed to be one its selling points, but my guess is that this would be a hard sell. 


Manufacturers may be deterred from producing FREEBOTS for other reasons too. They 
may, for example, fear being sued on a variety of grounds. Let's say you get attached to your 
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FREEBOT, but one day you lose your temper. As a result, the robot goes into pennanent 
nonresponsive mode. You are afflicted by grief. Can you then sue the robot company for 
emotional distress? 


Other possible disadvantages of FREEBOTS are that they may learn to use the threat of 
rejection to manipulate their owners. That is, they may come to use emotional blackmail in 
the same way that people often do. 


From this brief (and admittedly speculative) discussion of the pitfalls of both FREEBOTS 
and RELIABOTS, it seems that FREEBOTS have many of the potential downsides of real 
human relationships, so may lose much of their potential advantage over human 
Companions, while RELIABOTS have fewer disadvantages, but thereby lose some of their 
attractiveness. The problem at the heart of this dilemma does not lie with any technical 
limitations of the robots themselves, just as it does not lie with some contingent feature of 
human partners. It lies with our very desires themselves. We want contradictory things: a 
romantic partner who is both free and who will never leave us. TSA is therefore not correct, 
for it implies that the reason why human-human relationships often fail to live up to our 
expectations is that the humans we encounter are so often flawed, when in fact a more 
fundamental reason is that our expectations themselves are inconsistent. It may be easy to 
avoid the uncomfortable realisation that our desires themselves are to blame for our 
frustration, and blame our partners, when we do not have a way of correcting the supposed 
flaws we find in our partners. But if robots of the kind that Levy imagines were available, 
and those who sought solace in robot partners were forced to make a deliberate choice 
between FREEBOTS and RELIABOTS, it might be harder for them to persist in the illusion, 
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and they might be forced to accept that the source of their dissatisfaction lay partially inside 
them. Levy presents robot Companions as the best of both worlds. But in fact, they merely 
make the dilemma inherent in all relationships more visible. 


Love and/or sex 


Levy's book is titled Love and Sex with Robots. This paper is primarily concerned with the 
former - that is, with ongoing human-robot relationships involving mutual affection and 
cohabitation. Sex may or may not be a part of this relationship. Sex with robots outside of 
such a relationship would be akin to one night stands between humans, or sex between human 
prostitutes and their (human) clients. Prima facie, a version of TSA in support of GST that 
was restricted to such sexual encounters (outside of long-term romantic relationships) might 
be more plausible than the version of TSA/GST advanced by Levy, since the purely physical 
aspects of such encounters may dominate, or completely replace any second-order desires. In 
other words, the claim that non-romantic sex (“just sex”) with a robot may be more satisfying 
than non-romantic sex with a human might be more plausible than the claim that romantic 
relationships with robots may be more satisfying than romantic relationships with humans, 
given sufficiently advanced robots. TSA may be more applicable with this form of GST 
because, in this case, TSA can be restricted to purely physical features. To take the most 
extreme case, the client of a prostitute may have preferences about the prostitute's physical 
characteristics, but surely he does not have any desire that the prostitute freely chooses him. 
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Apparently, even this cannot be stated with confidence. There is some evidence that male 
use of prostitution depends in part on the client entertaining the belief (or fantasy) that that 
the prostitute has freely chosen to have sex with them (indeed, prostitutes try to mimic this) 
or at least that the prostitute has freely chosen her profession. When regular clients are 
forced to attend “re-education classes” that explain how pimps recruit, control, and exploit 
women and girls for profit, and learn about the links between local street prostitution and 
larger systems of human trafficking, these men tend to reduce their use of prostitutes (Shively 
et. al., 2008). This suggests that many clients not only believe that prostitutes choose to have 
sex with them, but also want this to be the case. So it seems that even here, where it is plain 
to most people that the prostitute does not choose her clients out of a desire to be with them, 
the clients prefer to fantasise that they have been chosen, and find that their enjoyment of sex 
with prostitutes diminishes if they cannot maintain this fantasy. 


It is hard to say how this observation bears on the ‘just sex’ fonn of GST. It could point 
either way. On the one hand, it might suggest that the objection to TSA outlined above 
would apply equally here, since people's enjoyment of sex, like their enjoyment of romance, 
involves second-order desires as well as first-order desires. On the other hand, it might 
suggest that it is remarkably easy for people to fool themselves into thinking that they have 
been freely chosen by their partner when in fact it should be clear that the partner has had 
very little, if any, choice in the matter. If this is true, then perhaps the dilemma outlined 
earlier is vastly overstated, and people will be very satisfied with RELIABOTS after all, since 
they will easily fool themselves into thinking that RELIABOTS have freely chosen to be with 
their owners. In this way, people really would get the best of both worlds - romantic partners 
who (appear to) freely choose them, and yet will never leave. People would gain this delight, 
of course, only at the price of deluding themselves. 
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Conclusion 


The ‘greater satisfaction thesis ’ (GST) states that romantic relationships with robot 
Companions might be more satisfying than relationships with humans. One argument that has 
been advanced in support of GST by David Levy is that people will be able to specify the 
features of robot Companions precisely in accordance with their wishes (which I call the 
“total specification argument”, or TSA). In this paper, I have argued that TSA is wrong, 
because it breaks down when we consider certain behavioral characteristics that we desire in 
our partners. We typically want our partners to choose us freely and therefore to be capable 
of rejecting us permanently. I call this the “free will objection” (FWO) to TSA. 


FWO is not decisive against TSA - it suggests difficulties with it, no more. Ultimately, GST 
is an empirical hypothesis which may become more or less likely as robots become more 
advanced, and we gather more evidence about human-robot relationships. 


Even if my argument against FWO turns out to be right, this would not show that GST is 
wrong. It would merely demolish one particular argument for GST. TSA could be wrong, 
and GST might still be correct. In particular, the version of TSA advanced by David Levy is 
intertwined with a series of other assumptions about the social context of partner robots (they 
are made in a factory according to specifications provided by the user; they are consumer 
products designed to serve the needs and wishes of humans; and are private property). In a 
different social context, TSA might not even apply. For example, if robots made copies of 
themselves, for their own purposes, and were not the property of anyone, and were 
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sufficiently advanced, they might enter into relationships with humans of their own free will. 
For all we know, it might well be the case that many humans would find relationships with 
these robots more satisfying than relationships with other people. However, this is not the 
kind of situation envisaged by Levy. 


This consideration suggests that predictions about human-robot relationships always need to 
be considered as more-or-less complete scenarios, in which a whole set of hypotheses is 
advanced jointly. These hypotheses always include auxiliary assumptions about the social 
and legal status of the robots in question, assumptions about the level of technological 
advancement, and other matters. The plausibility of any prediction about human-robot 
relationships can only be assessed in the context of the general scenario within which it is 
advanced. This suggests that the scholarly and academic treatments of this subject (such as 
Levy's) have much in common with fictional treatments (such as those of Philip K. Dick and 
Isaac Asimov). Indeed, the fictional treatments may often be superior, since they often take 
more care to make their assumptions about the social context explicit, and take more care to 
make the story internally consistent, and externally consistent with what we know about 
human nature. 
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Falling in love with a Companion - David Levy 


Abstract 


In 1984, in her groundbreaking book The Second Self, Sherry Turkle made us aware of the 
tendency of some people to develop relationships with their computers. Turkle described one 
such example, an MIT computer hacker who she called Anthony, who had “tried out” having 
girlfriends but preferred to relate to computers. I believe that the developments in AI since 
then have demonstrated a progression in human-computer relationships to the point where we 
can now say with confidence that, in the foreseeable future, significant numbers of Anthony’s 
and their female counterparts will be falling in love with software Companions. This position 
paper summarizes my arguments. 


In discussing this subject I have found a certain instinctive measure of disbelief from some 
people, often in the fonn: “If it isn’t human, how can you love it?” This scepticism can easily 
be refuted with two categories of counter-example—the love that many people exhibit for 
their pet animals and for their virtual pets. Some pet owners derive even more satisfaction 
from their pet relationships than they do from their social relationships with people. Love for 
a pet is often very strong, sometimes even stronger than a person’s love for other humans, 
and can be manifested in various ways: the offer of a reward for a missing animal; an owner 
bequeathing an extraordinarily large sum of money for the care of their pet, even making the 
pet a millionaire; spending considerable amounts on pet food, healthcare, and sometimes on 
their animal’s grooming and clothing; and interacting with their pets in ways redolent of 
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human-human relationships: giving them names, feeding them from their own plates at meal 
times, celebrating their birthdays, allowing them to sleep on the owners’ beds, talking to 
them, and considering them as members of their family. And when a pet dies, the intensity of 
the owner’s feelings of grief will often be very similar to those experienced with the 
bereavement of a spouse or partner. Sometimes an owner’s love is seen in bizarre news 
reports: a groom designating his dog as best man, a divorcing couple battling in the courts 
over the custody of their pet; there is even a web site (www.marryyourpet.com) that offers 
owners the opportunity to “marry” their pets for a suitable fee. Clearly, all of these 
manifestations of love are appropriate ways to treat human loved ones. 


In The Second Self, Turkle pointed out that Man’s relationship with animals has certain 
parallels with his relationship with computers: “Before the computer, the animals, mortal 
though not sentient, seemed our nearest neighbours in the known universe. Computers, with 
their interactivity, their psychology, with whatever fragments of intelligence they have, now 
bid for this place.” I therefore continue this discussion of emotional attraction by transferring 
its focus from pet animals to virtual pets, which are simply a fonn of computer. The human 
propensity for loving pet animals informs our understanding of the nature of human 
emotional attraction to virtual pets. 


For many of those who value their relationship with their pet animal more highly than their 
relationships with other humans, it would not be very surprising if a virtual pet or a robot 
were to be regarded in the same vein, supplanting other humans as the most natural objects of 
affection. Such a strength of affection was observed in millions of owners of the late 1990s 
toy phenomenon, the Tamagotchi. This virtual pet incorporates software that exhibits pet-like 
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behavior patterns. The owner must care for her Tamagotchi in its virtual world, by pressing 
buttons to simulate the giving of food and drink, the playing of games, and other behaviors 
typical of a mother-child relationship, ensuring that the Tamagotchi will survive and thrive. If 
the Tamagotchi is neglected it can “grow ill” and “die”, often causing heartbreak to its owner. 
The creature’s behavior patterns have been programmed to change with time, in order to give 
the owners the sense that each Tamagotchi is unique and therefore provides a unique 
relationship for the owner, just as each pet animal and each human are unique. 


The literature abounds with anecdotes about Japanese Tamagotchi owners who go to great 
lengths in order to preserve the life and well-being of their virtual pet—businessmen who 
postpone or cancel meetings so as to be able to feed their Tamagotchi and attend to its other 
essential needs at appropriate times; women drivers who are momentarily distracted in traffic 
while responding to the beeping of their needy electronic creature; a passenger who had 
boarded a flight but felt compelled to leave the aircraft prior to takeoff, and vowed never to 
fly with that airline again, because a flight attendant insisted she turn off her Tamagotchi, 
which the passenger felt was akin to killing it. Every example reflects the attitude of devoted 
Tamagotchi owners that their loveable virtual pet is alive. 


The effect of this craze in the late 1990s was to spawn a culture in which some electronic 
products are accepted as having life-like properties. In a 2006 article in the London Review of 
Books , Turkle explains that, as a result of this change in perception as to the aliveness of 
artefacts, “people are learning to interact with computers through conversation and gesture. 
People are learning that to relate successfully to a computer you have to assess its emotional 
state (...), you take the machine at interface value, much as you would another person.” 
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Hand-held virtual pets such as the Tamagotchi are the simplest form of the genre, based on 
low-cost electronics that allow a retail price of $15 or less. The next step up in complexity is 
the virtual pet that “lives” on the computer screen, i.e. a form of “Companion”. The most 
believable and lifelike of these characters exhibit a variety of social cues: intelligence, 
individuality, sociability, variability, coherence, and some conversational ability. When their 
conversational ability has improved, and when these virtual characters are also able to 
recognize the user’s emotional state by employing affective computing technologies, such as 
those being developed by Rosalind Picard’s group at MIT, they will become rather 
compelling. But already computers are increasingly being regarded as our social partners, and 
with the evidence amassed by Clifford Nass and his team at Stanford University, it is not 
difficult to understand why. Nass’s group has discovered, inter alia, that their test participants 
quickly came to regard a computer program as male or female, depending partly on the voice 
of the program’s speech synthesizer; and the participants tended to be polite in their 
interactions with the computers in the experiments. These behaviors were despite a uniform 
denial by the participants that computers can have gender, or have feelings, or that they 
deserve to be treated politely! 


Giving a computer the means to express appropriate emotional responses is a task that falls 
within the development of a software “emotion module”. The work of Scott Reilly and 
Jospeh Bates at CMU on the Oz emotion module, Juan Velasquez’ Cathexis program, and the 
work of Cynthia Breazeal’s group at MIT, are amongst the best known examples created to 
date. Research and development in this field is growing, within both the academic world and 
commercial software and robot manufacturers, and especially in Japan and the USA. I am 
convinced that, within twenty years at the latest, there will be artificial emotion technologies 
that can not only simulate a full range of human emotions and their appropriate responses, but 


150 



also exhibit non-human emotions that are peculiar to computers (and robots). Developments 
in this area will benefit from the extensive research that has been conducted into the social 
psychology of human-human relationships and human-human communication, research that 
is also relevant to human-computer relationships. For example, Timothy Bickmore and 
Rosalind Picard at MIT have found that people use many different behaviors to establish and 
sustain relationships with each other, and that most of these behaviors could be used by 
computer programs to manage their relationships with their users. It is true that relatively 
little progress has been made during the past half-century in the recognition and 
understanding of conversational language, and that these fields require quantum leaps in 
progress for the conversational abilities of computers to rise to the levels necessary to make 
computers appealing as conversation partners, or life partners, for humans. But such leaps 
will certainly come, partly through the emergence of much faster and bigger computing 
technologies that will enable the use of yet to be discovered software techniques—ones that 
are probably not possible with today’s level of computing power but which will be possible 
using the computers of the future. 


Why do I believe that these Companions of the decades to come, even with their amazing 
conversational skills, will be sufficiently enticing to cause large numbers of us to fall in love 
with them? My own research has pointed to ten major reasons that have been identified by 
research psychologists as contributing to the falling-in-love process between humans, almost 
all of which reasons could equally apply to falling in love with a software Companion (or a 
robot). One of these reasons, for example, is reciprocal liking — Peter is more likely to fall in 
love with Mary if Peter already knows that Mary likes or loves him; so a Companion 
programmed to exhibit love for its human will satisfy this particular criterion. Another reason 
is similarity — two people are more likely to fall in love with each other if they are similar in 
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some respects; so a computer endowed with knowledge about a user’s hobbies, and 
programmed with similar personality traits to the user, will again have a head start in its 
attempts to engender love from its human. 


Although the practice of treating computer characters as our social partners is still very much 
in its infancy, the concept is gaining currency as more and more interactive personalities 
appear on the market, as they become steadily more lifelike, more intelligent, more 
emotional, and more emotionally appealing. When Companions are able to exhibit the whole 
gamut of human personality characteristics, their emotional appeal to humans will have 
reached a critical level in tenns of attracting us, inducing us to fall in love with them, 
seducing us in the widest sense of the word. We will recognize in these Companions the same 
personality characteristics that we notice when we are in the process of falling in love with a 
human. If someone finds a sexy voice in their partner a real turn-on, they are likely to do so if 
a similar voice is programmed into a Companion. That the Companion is not in the physical 
presence of the user will become less and less important as its software becomes increasingly 
convincing. Physical presence was shown long ago to be a non-essential ingredient for falling 
in love—just think of the romances and decisions to marry that developed between pen- 
friends in the days when the postal system was the most popular method of communication 
between people in distant places, not to mention the much more prevalent cyber romances 
that today replace the pen-friend relationships of old. 


Now consider the following situation. From the other end of an Internet chat line comes the 
image of a Companion, a virtual friend. You hear the seductive tones of its voice, you smell 
its artificial bodily scent, and it is endowed with all of the artificially intelligent 
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characteristics that will be known to A.I. researchers two or three decades from now (or 


perhaps sooner). You sit at home, looking at this image, talking to it and savouring its 
fragrance. Its looks, its voice and its personality appeal to you, and you find its conversation 
simulating, entertaining and loving. Might you fall in love with this Companion? Of course 
you might. Why shouldn’t you? Pen friend relationships long ago established that people can 
fall in love without being able to see or hear the object of their love, so clearly, being able to 
see it, finding its looks to your liking, being able to hear its sexy voice, and being physically 
attracted by its simulated body fragrance, can only strengthen the love that you might develop 
in the absence of sight, sound and smell. 


And what will happen when this software technology, or even software at a much more 
primitive level of personality and intelligence, is integrated, via haptic interfaces, with 
dildonic devices such as vibrators and some of the more exotic sex machines now being sold 
on the Internet? The answer is obvious. We will not only be falling in love with software 
Companions, but also having sex with them. 
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Identifying your accompanist - Will Lowe 


Abstract 


What Companions are or should be will depend on what we want them to do for us. And this 
depends on what they can do for us. In the end, I shall argue that whether Companions should 
best be thought of, programmed and regulated as others, or as extensions of ourselves, is a 
tactical question. One path will simply be more effective than the other. Either way, it’s going 
to be all about us. To make a start let us switch to the second person: What can a Companion 
do for you? 


Computer scientists are designing digital Companions for people - that is, virtual 
conversationalists, or digital confidants. “They would [. . . ] build a life narrative of the 
owner. [. . . ] You could call it autobiography building for everyone, [ . . ] inner content 
could be elicited by long-term conversations ” (Wilks, this volume). They express a view 
about the nature of such Companions, the outcome of ex- tended interaction with such 
Companions, and the method of their interaction. In short, Companions are agent-like others; 
they help us understand ourselves; and they do so by way of conversation. But there is an 
obvious tension between the first and second parts - the mention of autobiography gives it 
away: Are these Companions truly others, or are they us? Rather than attempt to say whether 
they are, or should be thought of as other agents or as filtered versions of ourselves, I would 
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like to take one step back. 


We begin with a practical and not-to-distant use case: a Companion can remind you to take 
your medicine on time, and to ensure that you finish the course of pills. For medical issues 
this will be easier and work better if the Companion has access to your medical records. This 
is both empowering, because you are immediately more integrated in the project of 
maintaining your health, and controlling, because it ensures that advice you get from doctors 
cannot be so easily ignored. And it has significant social benefits, allowing large amounts of 
accurate epidemiological data to be harvested via your Companion’s other (one might hope 
suitably anonymised) interactions with those interested in public health, from both research 
and policy perspectives. Indeed, Companion-like projects already exist to keep the demented 
out of institutional care for longer by making their environments more ‘intelligent’ 32 . Other 
cognitive deficits may also be best addressed by a Companion approach: the National Health 
Service already makes use of computer-based cognitive behavioral therapy courses for 
phobias, panic attacks, and depression. Indeed, such approaches may work better with a 
Companion involved; it will have more information about your local environment to tailor 
your treatment and monitor your progress. More generally, the idea of an extended period of 
agent interactions in order to provide a framework for understanding your choices, 
obligations, and relationships could well be described as autobiography-building. It is also 
called therapy. One of the more optimistic possible roles for a Companion is to make you 
more rational. Romanticism aside, it is better and less frustrating if your preferences are 
ordered consistently over currently available goods, and over time 33 . Consider these two 
types of ordering. It is well known that psychological preferences are sensitive to the framing 
of choices. Usually, the framing is done by another, such as the store whose window you are 
looking into, or the state whose tax code you are attempting to negotiate. The advantage of a 
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Companion is that it can reframe your choice set, perhaps according to criteria that you have 
provided with it beforehand. The possibility of using a Companion as a level of indirection 
for choices is also helpful when the choices are difficult, because they require expertise you 
do not have, or because they are easy to make now, but hard to make later. This latter is a 
ubiquitous planning problem, referred to generically as the problem of inter-temporal choice. 
Whether it is the cream cake forbidden by your diet, the last week of antibiotic treatment, or 
contributions to your holiday fund, when the sirens of plan-breaking temptation sing, it may 
be your Companion that ties you to the mast. 


The problem of inter-temporal choice can be described by individual discounting curves: 
assume that for any good, the utility of having it today is greater than tomorrow, and still less 
the day after. Classically rational agents discount their utilities at a constant rate, e.g. every 
extra day spent waiting reduces utility by 10%. This generates a curve of expected utility that 
decreases exponentially towards the present moment from its maximum when the good is 
available. The exponential fonn guarantees that utilities retain their rank ordering; 
preferences over goods will not reorder themselves as the nearest one comes within reach. 
Most mammals, however, discount at a variable rate depending on the proximity of a good 
leading to a ‘hyperbolic’ discounting curve. This can lead to situations where the expected 
utility of a larger but later good is initially greater but then at a certain point less than that of a 
smaller sooner one, leading the latter to be chosen. Fortunately for planning, the curve traced 
by a composite good explicitly constructed by bundling together many lesser goods over time 
is more nearly exponential than the curve of any of its components 34 . While there is 
agreement on the form of the cognitive problem, diverse mechanisms have been suggested to 
explain it, including folk-psychological theories of‘will power’, religious theories of 
‘temptation’, and most recently in neurophysiological accounts 35 . For the purposes of 
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thinking about Companions, George Ainslie’s account is particularly suggestive 36 . 


Begin by noting the structural similarity between the problem of inter-temporal choice and 
the prisoners’ dilemma in game theory. In a one shot game the equilibrium policy is to defect, 
gaining more utility than the other player but less than if you both co-operated. Other policies 
are equilibria if the game is repeated. Concrete proposals usually suggest reputation (as a co- 
operator) as a mechanism to maintain long tenn optimal behavior despite the temptations of 
short term gains realised by defection. 


In Ainslie’s framework you are engaged in just such a repeated game with your future selves. 
If you can frame sets of goods that will occur at different times as parts of a larger good then 
discounting for the larger good will become more exponential, and thus less liable to be 
trumped by a sooner smaller rewards. Ainslie describes this cognitive process as bundling, 
typically a mixture of asserting equivalence across actions - all drinks are equally bad for me 
- and a personal rule - because I don’t drink. To the extent that breaking such a rule by 
succumbing to a nearby pint affects the probability that it will happen again, rule breaking 
incurs a reputation cost (your reputation to yourself). There is therefore always a motivation 
to interpret each rule breaking as a justifiable exception - drinking at a birthday - which in 
turn may perversely motivate systematic misunderstanding of oneself and the world. The 
dynamic of setting personal rules, sometimes breaking them, and then dealing with the 
consequences for your self image can therefore be described as inter-temporal bargaining. 


Ainslie describes the mechanisms of decision in an unaided agent, but the presence of a 
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Companion changes things. Crudely, a Companion may act as enforcer, distracting you from 
proximate but self-defeating pleasures, or telling you off afterwards. More interestingly, a 
Companion may simply inform you about what you’ve been doing, making it harder to 
maintain a transient pleasure-justifying narrative that you would be ashamed of later. 
Companions are also plausible repositories for personal rules that discourages opportunistic 
rewriting - a commitment mechanism similar to illiquid retirement savings plans 37 . A 
Companion may also independently assess certain predictive probabilities. 


This may be a problem: If it is necessary for maintaining your current course of action to 
believe that every ‘exception’ signals the end of your personal rule and triggers a steep 
reputation cost, then it may not be helpful to have a Companion inform you that there is in 
fact a 0.8 probability that your latest ‘lapse’ will not be repeated. Naturally, a sophisticated 
Companion would do its calculation conditioning on both the value itself and its general 
criteria for telling you, leading fairly directly to a version of Newcombe’s Problem 38 . In 
short, a Companion might help you make your choices in a place where “those sinkings of 
the heart at the thoughts of a task undone, those galling struggles between the passion for 
play and the fear of punishment, would there be unknown.” 


The previous passage is taken from Jeremy Bentham’s discussion of the Panopticon and its 
advantages as a school design 39 . The possibilities sketched above spell out the irreducibly 
Panoptical nature of Companionship. Our two possible Companion types appear as extremes: 
Either the Companion is another - e.g. the watching eye and punitive hand of the state, even 
if only its health service - or the Companion is a cooler headed version of yourself, a 
programmable super-ego. Interesting Companions no doubt lie between these extremes, but 
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the notion of inter-temporal bargaining brings out the common Panoptical core of each. This 
core promises (or threatens) to make you transparent to others, and to yourself It is 
conventional to deplore Bentham’s Panopticon, but the common elements of surveillance and 
intervention may be as desirable for constantly renegotiating the extended bargain of a 
coherent self as it is offensive when imposed by a state. The watchful Companion can exist at 
any point between, so its construction obliges us to be clear about the nature of the difference. 


While a Companion might make inter-temporal planning easier when all goods have been 
bundled and choices framed by increasing the information available to us about ourselves in 
choice situations, we have not addressed the question of how choices become framed to start 
with. Considering this prior process - which behaviors count as ‘my diet’? What constitutes a 
‘lapse’? - and the permanent possibility of on-the-fly redefinition demonstrates the limits of 
Companion-driven transparency. For example, the probability of not lapsing again is only 0.8 
under some particular bundling of goods and plans, some set of equivalence classes over 
actions. It will be different or perhaps undefined in other (potentially non-commensurable) 
decompositions of the social world. Every probability needs a sample space, and your 
Companion will partly constitute it. 


Let us return to the first example of this paper. You tend to accept the information and 
interventions of your doctor. This is partly because you share her assumptions, and partly 
because if you cease to trust her judgement you can find another doctor, resort to prayer, or 
switch to herbal remedies - each embodying a distinct conceptual scheme and rule set that a 
Companion could help you work within 40 . 
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Panoptical objections arise most quickly when your Companion is chosen for you, 
particularly by a state. This is reasonable. Not only is there no general reason to assume that 
state’s bundling of goods and plans is the best one for you, but states too have inter-temporal 
choice problems 41 . But problems also arise when you choose for yourself. A Companion can 
impose the thinking and values of an entire community on you, in a way that is unaffected by 
subsequent changes in that community’s thinking. For example, a Companion may still be 
‘helping’ you ward off spells when a ceasefire agreement has already been signed between 
the other members of your religion and modern medicine. This is problematic despite the fact 
that you chose your Companion freely, perhaps to bolster your faith. 


Returning to the original question: It is an empirical matter whether the inter-temporal 
bargaining we seem to need, to stabilise our choices and construct coherent selves, is best 
facilitated by Companions that are better infonned, cooler-headed reflections of ourselves, or 
by distinct agents interested in us in the same benign but instrumental way as our therapist or 
financial advisor. Either way, they will need to be distinct and independent enough for us to 
treat their advice, information, and representation of our rules as real constraints on our 
action, but sufficiently closely aligned and sensitive to our goals that we do not feel them as 
an imposition. Companions will inevitably structure our choices, but we will choose them to 
do precisely that. 
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Look, Emotion, language and behavior in a believable virtual Companion - 

Daniela M. Romano 

Abstract 

A good friend is the one that make you laugh, shares your deepest emotions and is there to 
listen and help you when you need it. Can we have we have a synthetic Companion with 
these qualities? He/she (we assume it has the status of a person at this stage) should be able 
understand and express emotions, to talk and listen, know who you are and what you feel and 
like. The Companion considered in this chapter is a synthetic creature living in a virtual 
world, able to display believable human-like qualities. The concept of believability within 
virtual environments is also discussed here together with some of the problems connected 
with the creation of such Companion. 


Introduction 

Many people around the world spend a lot of their time playing social games like The Sims 42 
and Second Life 43 that amuse, entertain, stimulate emotions and provide an opportunity to 
have real life-like experiences (Romano, 2009). What moves people to spend several hours 
of their time inside these virtual worlds is not the quality of the graphics or the naturalness of 
the interaction, but the social opportunities provided (Schroeder, et al, 2001; Brown & Bell, 
2004). People that never meet in the real world start social relationships, and live virtual 
experiences together. The major attraction of such cyber-spaces is the fact that there are 
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other people in there. People experiencing urban virtual environments focus more on the 
people populating them than the environments themselves (Romano, 2002). 


The presence of synthetic Companions enhances the user’s experience, facilitates the 
interaction with software and technology, and increases the perception that we are dealing 
with something more close to a real world situation than an artificial (Burkitt & Romano, 
2008). The presence of other people, although synthetic, encourages humans to interact 
within the virtual environment as if it was real, with rules similar to those established in the 
real world, but also with other that have been established in the new virtual dimension. 


A Believable Companion 

The concept of believability, as considered in this chapter, is synonymous with “social 
presence”, that is the sensation of being in a virtual environment with others, where the one 
has the feeling that s/he is interacting, through mediated communication, with an intelligent 
being (Biocca, 1997 ). In this context, the measure of effectiveness of social presence, or 
Companion believability, is the degree to which the user is able to perceive the intelligence of 
another (Biocca, 1997). Such intelligence is not just merely the ability to accomplish a task, 
which will not be discussed here, but also more importantly social and emotional intelligence. 


For social synthetic interactions to be believable the synthetic Companion needs to display 
emotional intelligence, which is displaying an intelligent behavior that is also emotionally 
consistent with the context, the actions and the emotion of the user. Emotions play an 
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important role in interaction: For example, we remember better those words that are 
pronounced with an emotional nuance and in a particular negative one. Also, we attach an 
emotional connotation to each experience of our lives, and we use it as a motivator for future 
actions and a trigger for memories that have been formed under similar emotional conditions. 


This Companion or 3D virtual character should also exhibit a full consistency across the 
senses if it is to make one feel it is human. Such believability does not always imply realism. 
Walt Disney cartoon characters exhibit human traits which are clearly recognisable and 
accepted as believable within an embodiment most often far from exhibiting a photorealistic 
look, yet they express clearly recognizable human traits. 


As Garau et ah, (2003) and Vinayagamoorthy, et al. (2004) have shown, having a 
photorealistic 3D character does not make the virtual world more credible, but other aspects, 
such as behavior and expressions, verbal and non-verbal, are as significant as the visuals. 
Consistency is the key to believability 


Believability needs to be broken into a set of characteristics that together contribute to create 
the Companion: Its voice and discourse abilities, the quality of its visual representation, the 
body language and the emotional expressions, its social abilities to simulate an understanding 
of the user’s emotions and actions, all compose the overall make up of the character and 
contribute to its believability. 


Other attributes that affect believability during social interaction with a virtual Companion 
are also the environment in which the interactions take place, the user engagement in the task 
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to be accomplished within the given application, and the level of interactivity possible 
between the human and the synthetically generated creature, which is here considered as a 
property of the technology that the user manipulates for the interaction to take place: the 
input and output devices. These attributes, although important, are not reviewed in this 
chapter where the essence of the character’s qualities is considered in absence of context; 
Here we analyze some important characteristics for a truly believable friend. 


A good friend at run-time 

We are used to the wonder of 3D graphics presented in the latest films and special effects, in 
which graphically generated characters move and talk like real people. They look and behave 
realistically as they are often modelled on real people, motion-captured movements, and are 
dubbed by real actors. The scenes are rendered after hours of computational modelling and 
the dialogues are scripted. 


A real-time Companion, by contrast, needs to be ready to interpret and respond on the fly, its 
behavior, movements and dialogues need to be computed at run-time, the interactions with 
the human friend are potentially infinite. When creating a virtual friend one has to balance the 
cost and effect of each feature introduced and trade those off that can be ignored so as to 
improve the one that sustains the suspension of disbelief. 
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Looking like a human 


The key for success for a virtual character is not necessary to look like a human. Cartoons, 
for example, still generate emotions and reactions in the human spectators. The success of a 
synthetic character is in its coherence. 


As some authors report (Garau et al., 2003; Vinayagamoorthy, et al., 2004), having 
photorealistic people does not make them necessarily more believable if their behavior, 
speech and language are not at the same level as the visual qualities. Any discordant element 
makes the character less believable. There is a theoretical contrast between the quality of the 
movement of a synthetically generated character and its visual aspect described as the 
Uncanny Valley 44 . Masahiro Mori (Mori, 1970; Mori, 2005) represented the emotional 
response to a synthetically generated character as a curve that rises up to a point and then 
descends rapidly when the character resembles a human being more and more, to reach the 
lowest peak called the Uncanny Valley and then rises again very rapidly. Consequently, one 
has to consider how human the synthetic Companion should look like and still be believable. 


Communicating like a human 

To build a Companion that communicates like a human one has to consider the aesthetic 
qualities of the character movements, its behavior and its ability to interpret and respond 
verbally and non-verbally to the human user. Humour and politeness make a character more 
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human-like, but these characteristics need not only to be expressed, but also to be interpreted. 
Gupta, Romano and Walker (2005) and Gupta, Walker and Romano (2007) have developed a 
dialogue generator able to express sentences at different level of politeness according to the 
relationship the agent has with the person it is talking to. 


Non- verbal communication is also important to induce empathy so as to truly be one’s best 
friend. In particular, the emotions are mainly expressed in a non-verbal manner. Shaarani & 
Romano (2006, 2007), and Romano et al (2005) have studied how different strengths of 
emotion should be portrayed in virtual characters and what are the characteristic of the body 
movement that make the recognition possible. 


Behaving like a human 

Visuals, communication and look are not enough if the synthetic friend is not autonomous 
and able to generate its own behavior. Consequently, a Companion needs to have a 
computational model that allows him/her to generate expressions, emotions and credible 
behavior. The main areas to simulate computationally are the interpersonal factors, the 
emotions and the personality. 


Kolb & Whishaw (1996) proposed that cognition and emotion are connected and vary 
together. Our emotion conditions our intentions and how memory and ideas are formed: For 
example, feeling anxious increases the speed of one’s behavior. Some emotions appear to 
have a purely social function, as they induce a reaction in the spectator of such an emotion. 
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For example, if someone is annoyed it automatically generates a sense of guilt in the 
spectators of such a display of emotion. To be believable, a synthetic friend has to be able to 
sense the emotional atmosphere and react accordingly. Romano et al (2005), Burkitt & 
Romano (2008) have created a computational model that drives a synthetic character called 
BASIC (Believable Adaptable Socially Intelligent Character) that is able to generate an 
emotional response to an emotional event in input that is in turn generated by direct 
interaction with another character. 


We do not yet have answers for all the questions posed here, but we will surely need a great 
deal of computational power to simulate all the above characteristics in one artefact that is a 
real-time and believable Companion. 
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New Companions - Alex Taylor, Anab Jain, Laurel Swan 


Abstract 


This chapter draws on advancements in Artificial Intelligence (AI) and robotics to question 
the orthodoxy of artificial Companions research. Two areas of work are outlined and used to 
suggest artificial Companions need not be restricted to simulacrums of humans or animals. 
First, it is argued the developments in AI have given rise to the prospect of very different 
kinds of machines, machines that are unlike humans or animals but that we may still want to 
form relationships with. Second, details are presented of a project exploring energy 
autonomous robots. Ecobot, an example of such a robot, is shown to exhibit unique 
characteristics that may afford new, distinctive forms of Companionship. Finally, a design 
concept of an autonomously powered, household radio is presented to illustrate how these 
new kinds of relationships might be investigated further. 


Introduction 

Working in various capacities with the Socio-Digital Systems 45 group at Microsoft Research 
in Cambridge (UK), we have over recent years contributed to a programme of work that has 
criticised the ideas of intelligence prevalent in technological imaginaries. Loosely associated 
with an area of research referred to as Human-Computer Interaction (HCI), we have given 
special attention to smart homes and argued that many of the underlying motivations driving 
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the smart-home vision sit uncomfortably with some of the practices we’ve observed in our 
various empirical investigations of home life (e.g., Taylor et al. 2007; Taylor & Swan 2005). 
We have reasoned that it’s not only a massive and probably intractable technological 
undertaking to get intelligent technologies to work in the ways promised, but also that the 
very promise is fraught with theoretical and ontological uncertainties provoked by long¬ 
standing debates around what it actually means for machines to have intelligence. 46 


As an alternative to building artefacts and environments with intelligence (as in visions of the 
smart home), we have aimed instead to demonstrate through a range of examples that 
information and communication technologies (ICTs) can be designed to enable people to act 
intelligently. In other words, we have aimed to re-direct the programmatic efforts of domestic 
ICT design away from the possibility of intelligent things and towards an idea that privileges 
human intelligence. 


Reflecting on our position, one resounding flaw that occurs to us is how the person on the 
street, if you will, regularly attributes life-like qualities to things and, in doing so, 
occasionally imbues them with intelligence. This is done without observable difficulties; 
people readily describe their household appliances as clever, smart and so on, and are seldom 
slow to refer to things as dumb or stupid. Although these phrases are not immediate 
equivalents to intelligence, they are suggestive of the idea, 47 and more importantly 
demonstrate the capacity and indeed willingness people have to attribute inanimate things 
with the ability to think—even when knowing full well they don’t. 48 
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Giving further thought the insightful points made by Edwards and Grinter (2001) on the 
piecemeal adoption of technology into our homes, the aversion we have had in our research 
to the idea of intelligent things pervading our environments now seems all the more 
restrictive. Thus, what we have begun to contemplate is the possibility of people gradually 
taking up seemingly inconsequential digital things that they see and treat as intelligent— 
intelligent not in the sense of human intelligence, but nevertheless intelligent in some casual, 
everyday sense of the word. Taking this possibility to its logical conclusion, it’s plausible to 
imagine many of us living in environments suffused with an ‘Artificial Intelligence’—living 
with artificial Companions, if you will, but having reached that point without any clear intent 
or thought. As Edwards and Grinter suggest, a home’s smartness may come about 
incidentally as increasingly advanced technologies are incorporated piecemeal into our 
homes. Arguably, we have already started down this road. The readiness of youngsters and 
adults alike to care for their Tamagotchi-like devices and the less pronounced but perhaps 
more significant uptake of relatively simple robots like iRobot’s vacuuming Roomba (see 
Forlizzi & DiSalvo 2006), lend support to a vision of the most rudimentary ‘thinking’ 
machines seeping into everyday life. 49 


What we want to suggest in this chapter is that this rethinking of intelligence—shifting our 
understanding of it from something considered an essential feature of being human to part of 
the ordinary repertoire we use to make sense of our interactions with things—raises some 
interesting prospects for artificial Companions. Most directly, we hope to demonstrate how 
such a shift opens up some different ways of thinking about Companionship with artificial 
agents and, more broadly, machines. So, rather than assuming an artificial Companionship is 
achieved by simulating the kinds of intelligence(s) we are familiar with, such as human or 
animal intelligence, we will suggest that viewing intelligence differently gives rise to 
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possibilities that are far more wide-ranging. In this light, the provocative question becomes 
one of what kinds of (artificial) things might we be willing to see as Companionable that are 
unlike anything we know today. 


To develop this line of inquiry, we have chosen, in the following, to consider developments 
in two areas, one in Artificial Intelligence and, the second, in robotics. First, we reflect on 
recent advancements in AI and suggest some new perspectives on intelligence promote a 
rethinking of the relations humans have with artificial agents. From this narrow focus on 
intelligence, we then broaden our outlook and consider an emerging research programme in 
robotics, investigating energy autonomy. We suggest that efforts to build robots that are able 
to power themselves by digesting organic material prompt further questions about machines 
and their potential roles as Companions. Finally, we also present one of our own design-led 
projects in which we have attempted to explore ideas of Companionship through a 
speculative energy autonomous device. We report on this early work to illustrate our own 
attempts to investigate new kinds of artificial Companions and the issues surrounding their 
use. 


Artificial Intelligence Revisited 


First, then, we want to suggest that some of the broad-sweeping developments in Artificial 
Intelligence (and, in a related way, robotics) over the last ten to fifteen years have 
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implications for how we might think of intelligence and consequently what we might 
consider a Companion. 


As has been well documented, AI has undergone significant and dramatic changes since the 
1950s when Newell and Simon (1963) laid the groundwork for the project. Good Old 
Fashioned AI (GOFAI), as it is now known in certain circles (Haugeland 1985), was based on 
the premise that human reasoning could be symbolically depicted, represented and modelled 
from the top down. Then at the heart of AI, this premise was a direct successor of the 
mechanical rule-following that Turing (1950) ascribed to both computing machines and 
humans (and specifically the human mind). An exemplar of the GOFAI position was, and still 
is, the artificially intelligent chess player designed at least in theory to incorporate a 
searchable model of all possible pennutations within a certain range of play. This brute-force 
approach eventually resulted in 1997 in IBM’s much heralded Deep Blue II (or Deeper Blue) 
beating the then World Chess Champion, Gary Kasparov, though the win was not without 
controversy (see Dreyfus & Dennett, 2005). The hardware and computational demands at that 
time, were extreme, though, even given the clearly bounded rules and constraints of a chess 
game. 


GOFAI has been heavily criticised not only for the substantial demands its approaches place 
on computing machinery but also for its inbuilt assumptions concerning the processes of 
human thought and action. Even in the subsequent but closely associated cognitive science 
programmes, serious doubt has been cast on the idea that thought is systematically and 
sequentially processed (Dreyfus 2007; van Gelder 1998), let alone the presumption that these 
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processes occur solely in the brain (Clark & Chalmers 1998; Hutchins 1995). Those levelling 
such criticisms have argued that machines like Deep Blue and its offspring are not so much 
indicative of a machine’s capacity to simulate human intelligence, as they are illustrative of 
the hard problems AI programmers face in achieving anything like the dynamic, moment-by¬ 
moment qualities of human behavior and reasoning observable in the world. 50 


In a similar vein, but from very a different perspective, Lucy Suchman’s now much cited 
book, Plans and Situated Actions (Suchman 2006), offered a thorough critique of intelligent 
systems. Through a set of in-situ experiments, she demonstrated how human behavior, even 
when planned, is an emergent phenomenon worked out on an ongoing basis. In other words, 
Suchman showed how a priori representations of planned actions, on which GOFAI is 
predicated, poorly map onto people’s real-world practices. We will return to this point in a 
moment. Here, we want to emphasise that the criticisms directed at GOFAI (as well as related 
shifts in funding) led to a good deal of introspection on behalf of the AI community (see 
Lighthill 1973). Researchers and academic departments distanced themselves from ambitious 
AI programmes, a move exemplified by the adoption of less grand terms like ‘machine 
learning’ and the use of AI’s tools to address more bounded problems such as image 
processing, computer vision and expert systems. 


From this period, a ‘New AI’ has taken shape (in some cases, explicitly building on 
Suchman’s ideas (Agre & Chapman 1990)). The progenitor of this turn, Rodney Brooks, has 
pursued “emergence” as a central theme in machine intelligence (Brooks 2002, 1999). 
Broadly, Brooks and his successors at MIT have sought to build systems with initially simple 
computational underpinnings that adapt and leam using bottom-up processes so that 
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intelligence ‘emerges’, as it were, when the technology is put into the wild. The key 
difference from GOFAI has been to eschew any effort to represent the world, a priori, and 
instead have the machines react to as well as leam from the situations they are placed in. 
Brooks originally posed his ideas in terms of primitive life forms, robotic insects, etc. 
(Brooks 1991) although, unsurprisingly perhaps, they have gone onto find form in human¬ 
like robots. Inspired by Brooks’ thinking, Kismet (Breazeal 2003), for instance, offers a 
mechanically sophisticated, although visually crude, example of a humanoid robot designed 
to learn and respond to emotional cues. 


Alongside these ambitious works, an assortment of smaller scale projects has built on the idea 
of emergence. Distinctive amongst these projects has been a move away from simulating the 
kinds of thinking, actions or motivations we would normally associate with being human. For 
instance, Frederic Kaplan and his colleagues at Lausanne’s EPFL robotics laboratory have 
experimented with the intrinsic motivation of robots (Kaplan & Oudeyer 2007; Oudeyer, 
Kaplan, & Hafner 2007). Using Sony’s robotic dog AIBO as a platform, Kaplan demonstrates 
how robots can independently develop their own behaviors when programmed with simple 
learning algorithms. Collaborating with interaction designers, Kaplan exhibited examples of 
AIBO (stripped of its commercial programming) gradually learning to move, ‘cycle’ (on a 
custom built bike) and ‘swim’ (in a specially designed swimsuit), and appearing to do so in 
idiosyncratic ways. Beyond the amusing results, a crucial aspect to work such as Kaplan’s is 
the difference in thinking about machine intelligence: this work indicates how machines can 
exhibit seemingly intelligent behaviors without having to replicate the mental processes of 
humans, or animals for that matter. 
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A new found pragmatism in AI has further contributed to this apparent rethinking. Side¬ 
stepping Brooks’ notoriously dogmatic position against representation (Brooks 1991) and the 
quagmire of arguments around intelligence, a stance has been adopted that combines 
elements of GOFAI and Brooks’ adaptive learning (e.g., Verschure & Althaus 2003). Seen as 
not mutually exclusive, the different approaches are used as alternative resources for design 
to produce ‘hybrid systems’. The consequence has been to promote a more experimental use 
and application of machine intelligence, where new rather than rehearsed ideas of behavior 
and intelligence are pursued. 


The point of this potted history is that there are interesting lessons to be learnt from the 
circuitous route travelled from the brute-force machines of the GOFAI era to the robots and 
agents we see today. What we want to suggest is that the developments in AI have been 
instructive on two broad counts. First, and putting to one side the ambitious projects from AI 
researchers such as Brooks (2002) and Grand (2003) (also see Castaneda & Suchman), the 
efforts from within AI and robotics to build simple-functioning, autonomous machines that 
are responsive to situational variables, indicates a shift in thinking about intelligence. No 
longer is it exclusively assumed that the sort of intelligence to be attained in machines should 
simulate human intelligence. More subtle ideas are finding a foothold that are leading to 
machines that are merely suggestive of intelligence. Key here is that elements of AI and 
robotics are offering a path away from rigid and restrictive notions of intelligence in 
machines and towards alternative and in some cases new possibilities. 


The second lesson comes more from outside AI—broadly from an ongoing and maturing 
critique of AI’s developments—and provides us with an interesting possibility for 
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(re)framing the ideas we have of intelligence. As it happens, Suchman (2007) has played an 
important if not key role in this critique (Sengers, forthcoming). Suchman’s thesis criticised 
some of the assumptions implicit in intelligent or expert systems, specifically those being 
designed into the photocopiers her organisation manufactured. As we noted, a primary lesson 
those in the area of Human-Computer Interaction have drawn from this work has been to 
recognise the situated character of human action—that even planful behavior is contingent on 
the contexts we find ourselves in. Suchman, though, had another major theme to her work 
that was given far less attention (although one she has since pursued and elaborated on in her 
books second edition). As part of her critique, Suchman articulated an argument that 
foregrounded the constitutive nature of the human-machine intersection. That is, how the 
ideas of both human and machine are features of the work invested in construing the interface 
between them. She suggested that even the idea of human-computer (inter)action (re¬ 
configures how humans and machines are understood. 


Suchman’s arguments parallel and in some cases build on a programme of work—if it can be 
called that—that has re-cast or reconfigured the ontological distinctions made between 
humans (as well as animals) and things. Latour (1993) and the Actor Network Theory corpus 
(Law & Hassard 1999) offer contemporary examples. Especially relevant to our ongoing 
research are works around the theory of extended mind (specifically Andy Clark’s extensive 
contributions, e.g., Clark & Chalmers, 1998) and the feminist techno-science programme 
spearheaded by Donna Haraway (1991) as well as cultural studies personalities like Katherine 
Hayles (1999). Like the work from within AI, these positions although by no means unified, 
provide a way of re-imagining what machines might do (and the intelligence they might 
have). 51 More importantly though, they encourage us to rethink intelligence itself. They have 
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us reflect on not only the figurations (Haraway 1991) of human intelligence, but also to take 
seriously the prospect of alternative and possibly new conceptions of intelligence. 


At first sight, these developments in AI and subtly different perspectives on intelligence 
might seem some way from the prospect of artificial Companions. Yet it is worthwhile 
considering what these foundational changes in AI, on the one hand, and ways of thinking 
about intelligence as something constituted through human-machine interaction, on the other, 
could mean for our relations with artificial Companions. We want to suggest they point 
towards two relevant implications. 


First, it would seem there are a new wave of AI machines—founded on the move towards 
emergence but also including hybrid systems—that provide a compelling starting point for 
rethinking our Companionship with artificial agents. What we have is the possibility of 
artificial Companions that might originate as very primitive systems, but adapt or emerge 
over time in response to their environments, including our interactions with them. Thus our 
aim might not only be to build computationally sophisticated simulacrums of pet or even 
human agents, but also to investigate what the building blocks might be for artificial 
Companions that co-evolve with us in strange and unfamiliar ways. 


This closely relates to our second implication. The idea that intelligence is something that can 
be actively construed in things, invites us to wonder whether we might be open to a quite 
different possibility of intelligence in machines. Might we come to think of machine- 
intelligence as a new category of thing? For artificial Companions, this lends further support 
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to the possibility of rethinking what Companionship might be: if we are willing to see new 
forms of intelligence in machines, might we also be open to finding them Companionable in 
different ways? Might the unique properties of the machine—how it is seen to (inter-)act in 
the world—be enough to draw us into new kinds of Companionable relations? 


Energy Autonomy 


To develop these points and look beyond the specific questions surrounding intelligence, we 
want to draw attention to a small programme of work in robotics investigating the idea of 
energy autonomy. Although this work deviates somewhat from the kinds of AI research 
discussed above, what we hope to show is how a particular set of investigations into agent 
autonomy introduces some different, but still compelling possibilities for rethinking 
Companionship. 


The project we want to give specific focus to is based at the Bristol Robotics Lab (BRL), in 
the UK, and centres on a robot, EcoBot, that incorporates its own mechanisms for producing 
and utilising energy. The BRL see this “energy autonomy” as a crucial element of building 
self-contained robots that are independent of human supervision (Ieropoulos, Greenham, & 
Melhuish 2003). Thus the Ecobot project, that began in roughly 2002, has sought to build 
simple mechanical robots that can independently collect and metabolise organic materials in 
order to produce energy. 

Microbial Fuel Cells (MFCs) are central to this process of autonomous energy production. 
Briefly, MFCs rely on the breakdown of organic material by a microbial substrate to convert 
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bio-chemical energy into electrical energy. The MFC’s substrate, that metabolises the organic 
material, can consist of sludge, not dissimilar to that found in the common, garden-variety 
pond. The organic material fed into the cells might be a simple compound or something more 
complex. For example, the first of the project’s working robots, EcoBot I (Fig. la), used 
simple, refined fuels (i.e., sugar) to produce power, whereas the subsequent EcoBot II (Fig. 
lb) and III (Fig. 2) have been able to utilise more complex raw materials such as fruit, 
vegetables or even insects. BRL has been instrumental in improving the power yield 
efficiency of these fuel cells and also pioneered their use in robots. 




@@ Insert Figure 01 here 


Figure 1 . (a) EcoBot I and (b) EcoBot II (With pennission from the Bristol Robotics Lab). 


The successive incarnations of the EcoBot have also experimented with how the different raw 
materials are collected and fed into the MFCs. EcoBot I relied on sugar being manually 
supplied or ‘fed’ into an array of eight microbial fuel cells incorporated into the robot’s foam 
body (Fig. la). The electric current produced using the cells was subsequently used to (very 
slowly and erratically) move the robot towards light (i.e., phototaxis) to demonstrate its 
capacity to perform a simple task. EcoBot III has not only enabled the use of raw organic 
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materials, specifically flies, it also incorporates a synthetic digestive system that collects food 
(using a Venus flytrap like mechanism, Fig. 2b), channels it to the array of fuel cells, and 
either reuses or evacuates the waste. This latest version of the robot is thus seen by BRL as 
their first self-contained, energy autonomous system. 




Figure 2. (a) Base of EcoBot III and (b) top (with pennission from Bristol Robotics Lab). 


As noted, one motivation for the EcoBot project has been to build machines that are 
autonomous not only in sensing and moving, but also in how they power themselves. For the 
purposes of building fully independent robots, this motivation is understandable. However, 
and as the project’s members explain on their website, there are arguably more significant 
implications: incorporating the search for and consumption of‘food’ into a robot’s 
“behavioral repertoire” introduces “a paradigm shift in the way action selection mechanisms 
[in robots] have been designed so far.” In other words, the prospect of robots locating their 
own food and consuming it, autonomously, has the potential to radically alter the design of 
robots, how they operate and their use. 
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It is this arguably more fundamental perspective on Ecobot that we believe has direct 
implications for artificial Companions. Ecobot offers an example of the coupling that can 
occur between the technical and material developments of seemingly autonomous machines, 
and the potential this has to shape how we think of and interact with them. The realms of 
construction (e.g., science, engineering, design, etc.) and use appear to blend together in such 
a way that they unsettle our ordinary ideas of machines. In a similar way to the new wave of 
AI machines, robots like Ecobot appear to open up the possibilities for new kinds of relations 
with machines. 


Even superficially, EcoBot Ill’s curious assemblage of computation, electronics and 
material—both naturally occurring and fabricated—gives it a strikingly different feel to the 
things we normally associate with machines. It diverges, substantially, from the often touted 
humanoid or pet-like robots and yet hints at something more than merely a mechanically 
operated machine. The EcoBot’s ‘feeding cycle’, for instance, is self-regulated and has a 
direct impact on its own movements. Moreover, its mechanical outputs (i.e., movements) are 
exceptionally slow and occur in short bursts to allow sufficient charge to be accumulated via 
the MFC arrays. This aptly named “waiting behavior” (Ieropoulos, Greenham, & Melhuish 
2003) stands in stark contrast to the smooth, continuous movements that much of the robotic 
community aspires to in their systems. Admittedly, the EcoBot is only a proof of concept and 
any workable solution is likely to take a very different form. Nevertheless, as it stands, the 
robot expands our imaginations to contemplate a range of human-machine interactions that 
are altogether different. 
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Returning to the theme of artificial Companionship, we want to suggest that the questions 
raised around human-machine interaction by Ecobot also have implications for how we might 
think of machines as Companions. The kinds of processes incorporated into EcoBot’s design 
are, for example, likely to unsettle the simple distinctions we make between mechanical and 
living organisms; because of its bio-based digestive system, there is the inkling of the EcoBot 
incorporating an autonomy that is not programmed into it, but is, in a sense, a ‘naturally 
occurring’ phenomenon (i.e., digestion). Introducing this ambiguity problematises our 
relationship with the machine: should we feel obliged to feed it? If unfed, might we feel 
sympathetic towards it? Might its movements be read as distinctive mannerisms? Our point 
here is not to make any strong claims about the lifelike or even intelligent qualities of a 
machine like EcoBot or its offspring. Indeed, we appreciate it takes some if not a good deal 
of imagination to think of EcoBot as intelligent in any way. What such examples help us to 
consider, however, are the properties a robot might exhibit for us to imagine some possibly 
new forms of human-machine Companionship. 


Speculative Designs 


Over recent months, we have undertaken various design-led interventions in order to further 
investigate some of the themes discussed so far. Specifically, we’ve used a speculative design 
approach to examine what it might be like to live with some of these fledgling and still 
unfamiliar technologies in everyday contexts. 52 Our overarching aim has been to conceive of 
designs that invite a degree of curiosity in the observer and raise questions around the 
qualities a machine might exhibit for it to be thought of, in the vaguest way, as a Companion; 
our interests lie in machines that we are not immediately able to classify—that lie somewhere 
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in between known categories—but nevertheless might provoke some new and alternative 
ideas of human-machine relations. For this reason, we’ve sought to avoid using 
anthropomorphic or zoomorphic cliches, and to instead work on exposing some of the 
distinctive qualities of computation and energy autonomy that we have seen above. 

One example of these efforts is a microbial fuel celled radio (Fig. 3). The radio is a 
conceptual design for a household appliance powered by microbial fuel cells. A household is 
required to feed the radio’s cells with organic materials to ensure its continued operation. 
Rotten food, insects, and other forms of naturally occurring organic material are placed into 
the funnel at the top of the radio and channelled into the array of MFCs. The cells contain a 
microbe-rich sludge, and the organic matter used to feed the microbes passes through system 
and is recycled if possible. The radio’s housing is large to accommodate the MFC’s necessary 
to produce a continuous and readily available power supply. Any biodegradable waste is 
evacuated into the removable, lower chamber. 
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@@ Insert Figure 03 here 


Figure 3. Radio concept based on microbial full-celled power supply. 


One aim of this concept is to provoke questions around energy autonomy by incorporating 
MFCs into an appliance designed for an everyday setting. Clearly, the radio reveals that a 
degree of care is needed to keep the system operating. The MFC’s microbes must be kept 
alive for the cells to produce power, and any waste must also be managed. In addition, the 
radio must be regularly supplied with organic material for it to operate on demand. If the cells 
are under-fed, the quality of reception decreases and the power supply can become 
intermittent. The autonomous production of energy is thus set against the need for the regular 
supply of ‘food’ and cleaning. Hence, the radio’s basic operation is meant to have people 
speculate on the bounds of an energy autonomous device—what, exactly, is autonomous in 
the device and what if any new co-dependencies emerge from such an autonomy? 

The functional demands to maintain the radio are also meant to provoke further queries. If 
households must perform this care for the radio over time and tend, if you will, to the radio’s 
workings and performance, might this have some impact on their long-term relationship with 
the device? Might there be some gradual change to how one relates to a machine if it must be 
looked after? This relationship is accentuated by a further characteristic of MFCs. The 
microbial substrate of the MFCs can be conditioned over time to operate most efficiently with 
particular sources of energy. This conditioning is related, in part, to the history of the 
materials the cells are fed as well the environmental conditions the cells are exposed to. Thus 
the radio’s life-cycle and performance interleaves with its usage patterns and the timeframe 
of use. Over the radio’s lifetime, users must be attentive to what they feed it and the 
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conditions in which it is kept. In turn, the radio’s cells develop their own unique 
characteristics. As a result, the interactions with the radio are not just discrete, but extend to 
form an evolving, long-tenn co-dependency of sorts. 

We want to take care here in claiming a microbial fuel celled radio could lead from a reliance 
on human intervention to arouse a sense of Companionship. It would be fair to argue that 
households might be distinctly put-off by the prospect of something else in their homes that 
must be cared for. Our interest is not in whether such a radio would be desired, however, but 
rather if people might begin to see it in a similar way to some of the other kinds of things that 
require an obligation of care in home life. Might the radio be thought of as something 
different and yet still seen alongside plants or even pets? A reaction such as this would 
suggest that there maybe new forms of relations to be had with machines, and possibly the 
potential for different kinds of Companionship with them. 

Our attempts to answer these questions have so far involved exhibits of the radio (alongside 
other speculative designs; Jain & Taylor 2008), presentations to design and scientific 
communities, and visits with the device to a small number of households. Rather than report 
on the results of these excursions, our hope here has been to demonstrate how we have been 
building on some of the points discussed and sought to investigate a number of abstract ideas 
in concrete ways. 


Conclusions 


A simple response might be to trivialise projects like ecobot and even the disciplinary shifts 
in perspective in AI. After all, we have seen technological developments come and go, and 
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yet who we form Companionships with and how we form them have arguably changed little. 
A closer inspection of some of the advances in science and technology, however, do seem to 
indicate some pretty fundamental and possibly radical changes to the nature of things. From 
robots with simple, independently leamt behaviors and machines that are self-powered to 
biologically inspired software (Thrift 2004), and, indeed, technology inspired biology, things 
appear to be incorporating forms of autonomy that were previously unimaginable. Hybrids of 
living organisms, computation, mechanical operation, etc. are giving form to what feels like a 
new breed of things with their own distinctive motivations and behaviors. 

Attempting to avoid the regular hyperbole associated with such technological innovations, 
our aim has been to show that even the most tentative of beginnings in contemporary research 
are serving to challenge the preconceptions we have of things. What we hope to have 
demonstrated is that this potential for change to the nature of things opens up a space for re¬ 
imagining our relationships with machines. If things are on the cusp of a new lease on life, so 
to speak, why, we have to asked should we be bound to old ideas of Companionship? 

Humans and animals have been pretty good at being, well, human and animal Companions. 
So, could we ask what do things—things built on the new wave of scientific and 
technological advances—have to offer that might be different? As we made clear at the 
beginning of this chapter, a new class of AI machines have given rise to some very different 
ideas of intelligence. No longer is intelligence something we must judge against some 
measure of being human. Instead, we find we are able to construe forms of intelligence that 
are intimately associated with machines and their distinctive characteristics. In a similar vein, 
might we ask the same of Companionship? Could our relations with artificial Companions be 
transformed? In other words, at issue is not whether machines can be designed to be like the 
Companions we know, but whether machines might change how we think of Companionship. 
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On being a Victorian Companion - Yorick Wilks 


Abstract 

I have argued or suggested: 


• English Common Law already, in dogs, has a legal category of entities that are 
not human but are in some degree responsible for their actions and have 
“characters” that can be assessed. 

• Users may not want Companions prone to immediately expressed emotions 
and a restrained personality, like a Victorian Lady’s Companion, might 
provide a better model. 

• Language behavior is a complex repository of triggers for emotion, both 
expressed and causal, and this is often under-rated in the world of ECA and 
theories of emotion based on them. 

• Companion-to-Companion communications will be important and helpful to a 
user, and there is nothing in principle to make one believe that “secrets” 
cannot be handled sensitively in such an environment. 

• It is easy to underestimate the role of a user’s preference in selecting the 
personality appropriate to a Companion: it is not even clear that users want 
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Companions to be polite or agreeable—it may depend on personal choice or 
their functional role. 


• For many it may be appropriate for a Companion to become progressively 
more like its owner in voice, face, personality, memories etc.—exaggerating 
the way dogs are believed to adapt to owners—and if and when this becomes 
possible, for the Companion to become a self-avatar of its owner, there may 
well be other unseen consequences after the owner’s death. 


A colleague in this book remarks that James Boswell was a clear case of the inaccurate 
Companion: his account of Johnson’s life is engaging but probably exaggerated, yet none of 
that now matters. Johnson is now Boswell’s Johnson, by and large, and his Companionship 
made Johnson a social property in a way he would never have been without his Companion 
and biographer. This observation brings out the complexity of Companionship, as opposed to 
a mere amanuensis or recording device, and its role between the merely personal and the 
social. The last decade of the World Wide Web has shown extraordinary and improbable 
shifts along this axis: FLICKR, as we all know by now, began as a personal service for photo 
image storage, but has become a major social force in its own right, and the source of vibrant 
social groupings. 


The first Artificial Companion is, of course, Frankenstein’s monster in the 19C; that creature 
was dripping with emotions, and much concerned with its own social life: 
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Shall each man," cried he, ''find a wife for his bosom, and each beast have his mate, 
and I be alone? I had feelings of affection, and they were requited by detestation and 
scorn. Man! you may hate; but beware! your hours will pass in dread and misery, and 
soon the bolt will fall which must ravish from you your happiness for ever. (Shelley, 
1831, Ch. 20) 


This is clearly not quite the product that any modern Companions project is aiming at but, 
before just dismissing it as an “early failed experiment”, we should take seriously the 
possibility that things may turn out differently from what we expect and Companions, 
however effective, may be less loved and less loveable than we might wish. As Will Lowe 
points out in this volume, a fitness trainer is also a Companion, one whom we actually hire to 
be mildly cruel to us: No Pain No Gain, as they say in the gym. 


My interest in this issue is not a literary or theoretical one, but purely practical, and focused 
as the coordinator of a large EU-funded project committed to the production of demonstrable 
Companions, capable of some simple understandings of their owners’ needs and wishes. 

After two years into that project we do have something practical to show at 
(www.Companions-project.org) but not enough to be deployed, and in any case we have not 
yet explored sufficiently deeply the issues raised by Alan Newell and other contributors about 
how we actually find out what kinds of relationship people really want with Companion 
entities, as opposed to technologists just deciding a priori and then building what they want. 
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It is no longer fashionable to explore a concept by reviewing its various senses, though it is 
not wholly useless either: when mentioning recently that one draft website for the 
Companions Project had the black and pink aesthetic of a porn site, I was reminded by a 
colleague that the main Google-sponsored Companions site still announces “14.5 million 
girls await your call” and it was therefore perhaps not as inappropriate as I had first thought. 
Yet, for many, a Companion is still, primarily, a domestic animal, and it is interesting to note 
the key role pet-animals have played in the arguments in this volume on what it is, in 
principle, to be a Companion: especially the presence of the features of memory, recognition, 
attention and affection, found in dogs but rarely in snakes or newts. Children, of course, have 
other Companions, imaginary and stuffed, but here we will stay with adults for the moment. 


I would also add that pets can play a key role in arguments about responsibility and liability, 
issues also raised already, and that dogs, at least under English common law, offer an 
example of an entity with a status between that of humans and mere wild animals: that is, 
ferae naturae, such as tigers, which the common law sees essentially as machines, and 
anyone who keeps one is absolutely liable for the results of its actions. Companions could 
well come to occupy such an intermediate moral and legal position (see Wilks & Ballim, 
1990), and it would not be necessary, given the precedents with pets already available in law, 
to deem them either mere slaves or the possessors of rights like our own. Dogs are treated by 
English courts as potential possessors of “character”, so that a dog can be of “known bad 
character”, as opposed to a (better) dog acting “out of character”. There is no reason to 
believe that these pet precedents will automatically transfer to issues concerning 
Companions, but it is important to note that some minimal legal framework of this sort is 
already in place. 
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More seriously, and in the spirit of a priori thoughts (and what else can we have at this 
technological stage of development?) about what a Companion should be, I would suggest we 
could spend a few moments reminding ourselves of the role of the Victorian lady’s 
Companion. Forms of this role still exist, as in a recent web posting: 

Companion Job 

Posted: October 5, 2007, 01:11 AM 

I Am a 47 year old lady seeking a position as Companion to the elderly, willing to work as 
per your requirements. I have been doing this work for the past 11 yrs. very reliable and 
respectful. 

Location: New Jersey 
Salary/Wage: Will discuss 
Education: college 

Status: Full-time 

Shift: Days and Nights 


But here the role has become more closely identified with caring and the social services than 
would have been the case in Victorian times, where the emphasis was on company, 
preferably educated company and diversion, rather than care. However, this was not always a 
particularly desirable or even tolerable role for a woman. Fanny Bumey refers to someone’s 
Companion as a ‘toad-eater’ which Grose (1811) glosses as: 
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A poor female relation, and humble Companion, or reduced gentlewoman, in a great 
family, the standing butt, on whom all kinds ofpractical jokes are played off, and all 
ill humours vented. This appellation is derived from a mountebank’s servant, on 
whom all experiments used to be made in public by the doctor, his master; among 
which was the eating of toads, formerly supposed poisonous. Swallowing toads is 
here figuratively meant for swallowing or putting up with insults, as disagreeable to a 
person of feeling as toads to the stomach. 

But one could nevertheless, and in no scientific manner, risk a listing of features of the ideal 
Victorian Companion: 

• Politeness 

• Discretion 

• Knowing their place 

• Dependence 

• Emotions firmly under control 

• Modesty 

• Wit 

• Cheerfulness 

• Well-informed 

• Diverting 

• Looks are irrelevant, but must be presentable 

• Long-tenn relationship if possible 

• Trustworthy 

• Limited socialization between Companions permitted off-duty. 
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The Victorian virtue of Discretion here brings to mind the “confidant” concept that Margaret 
Boden (in this volume) explicitly rejects as being a plausible one for automated Companions: 

Most secrets are secret from some HBs [Human Beings] but not others. If two CCs 
[Computer Companions] were to share their HB-users ’ secrets with each other, how 
would they know which other CCs (i.e. potentially, users) to ’trust’ in this way? The 

HB could of course say "This is not to be told to Tommy". . but usually we regard it 

as obvious that our confidant (sic) knows what should not be told to Tommy, either to 
avoid upsetting Tommy, or to avoid upsetting the original HB. How is a CC to 
emulate that? 

The HB could certainly say "Tell this to no-one", where "no-one" includes other CCs. 
But would the HB always remember to do that? 

How could a secret-sharing CC deal with family feuds? Some family websites have 
special functionalities to deal with this. E.g. Robbie is never shown input posted by 
Billie. Could similar, or more subtle, functionalities be given to CCs? 


I think Boden brings up real difficulties in extending this notion to any conceivable computer 
Companion, but I do not think the problems are all where she thinks. I see no difficulty in 
programming the notion of explicit secrets for a Companion, or even things to be kept from 
specific individuals (“Never tell this to Tommy”), and Facebook already allows this. 
Companions will have fewer problems remembering to be discreet than people do, and I 
suspect there is less instinctual discretion than Boden suggests: people have to be told 
explicitly who to say what to in most cases, unless they are told to tell no one. In any case, 
much of this will be moot because Companions will normally deal only with one person— 
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which is what makes their speech recognition problem so much easier—they are trained for a 
single speaker—except when, say, making phone calls to an official, friend or restaurant, 
where they can try to keep the conversation to limited replies they can be sure to understand. 
The notion of a stored fact that must not be disclosed is simple to code, and the issue is wider 
in that the same fact must, to preserve the secret, not take part in inference processes either. If 
it is a secret that Tom is really a Russian, then the Companion should not do inferences like 
[IF X is of nationality Y THEN X will normally speak Y] and come out with an utterance like 
“I assumed Tom could speak Russian”, which would rather give the game away via the 
reverse inference, in the hearer [IF X speaks Y THEN X may well be of nationality Y]. 


The interesting case Boden raises is that of Companions talking to each other, and this was 
presumably always a risk for Victorians: that their human Companions would gossip behind 
their backs just as, in a slightly later age, Jeeves always chatted with other gentlemen’s’ 
gentlemen when the plot required him to. For our Companions this seems a positive 
development that we would in many ways encourage: imagine the shy older person in a care 
home, to shy too approach another for a lunch together. This would be something best settled 
between their Companions , each knowing the tastes and habits of their owner, to whom the 
“date” could be presented as a fait accompli. Again, many Companion-to-Companion 
interactions will be between an individual’s Companion and some fonn of “public 
Companion” such as one that takes restaurant bookings based on a user’s tastes; or at a 
hospital where a hospital-Companion could triage incoming patients, who may not be 
articulate about their condition, on the basis of detailed knowledge of the user’s medical 
records. When traveling, this Companion-to-Companion interaction in, say, a hospital could 
also combine with translation where the respective Companions worked out how to 
communicate across a language barrier. 
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In all these cases, Companion-to-Companion communication could be of obvious benefit to a 
user even if confidential information was at risk of disclosure: the user might have said 
“Never tell anyone I’m HIV positive” but in the hospital environment that constraint should 
obviously be overridden and the user’s condition revealed. One could say at this point that 
secrets may be relative to a situation and that there may be nothing more complex in a 
Companion’s guardianship of secrets than there is in explicit restrictions one could give to 
human hearers. The ultimate revelation of secrets by a Companion after a user’s death is a 
wholly separate and complex subject. There are already on the market (e.g. Deathswitch: 
http://www.deathswitch.com/) products that save and reveal passwords and ultimate letters 
and secrets at death; this is undoubtedly an area with enormous possibilities as the Internet 
makes actual death less apparent and immediate in the electronic world than it is the real one 
(see also Wilks http://people.oii.ox.ac.uk/yorick/2007/01/24/death-and-the-internet/ ). 


If the Victorian list above is in any way plausible, it suggests an emphasis rather different 
from that current in much research on emotions and computers (e.g. the HUMAINE network 
at emotion-research.net) and their possible embodiments and deployments to a public. The 
emphasis in the list is on what the self-presentation and self-image of a possible, and 
tolerable, Companion should be; its suggestion is that overt emotion may not be what is 
wanted at all. I have never felt wholly comfortable with the standard Embodied 
Conversational Agent (ECA) approach in which, if an avatar “has” an emotion, it 
immediately expresses it, almost as if to prove the capacity of the screen graphics. This is 
exactly the sort of issue tackled by Darwin (1872) and such overtness can seem to indicate 
almost a lower evolutionary level than one might want to model, in that it is not a normal 
feature of much human interaction. The emotions of most of my preferred and frequent 
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interlocutors, when revealed, are usually expressed in modulations of the voice and a very 
precise choice of words, but I realize this may be just cultural prejudice. 


On the other hand, pressing the pet analogy might suggest that, if that is to be the paradigm, 
then overt demonstrations of emotion are desirable and sought by pet owners: dogs do not 
much disguise their emotions, and their positive emotions are often welcomed by owners. 
Language, however, does disguise emotion as much as it reveals it, and its ability to please, 
soothe and cause offence are tightly coupled with linguistic expertise—as opposed to the 
display of gestures and facial expressions— as we all know with non-native speakers of our 
languages who frequently offend, even though they have no desire to do so, and often have 
no awareness of the offence they cause. What name to call someone by, or whether or not to 
use vocatives like “Sir”, “Mister”, ’’Miss”, “Missus” are enormously complex matters, kn own 
intuitively to native speakers but not to outsiders, who are never taught them and have 
nowhere to go for advice or instruction. These are not cultural matters across space only, but 
also time: it was pointed out long ago that in the 19C male Cambridge undergraduates would 
walk arm-in-arm and call each other by their last names, without giving offence, whereas in 
the latter part of the 20C they would use first names —since last names would have given 
offence —and never be seen arm-in-arm! 


All this could suggest that there is some confusion in the literature at the moment on what the 
desirable self-presentation of a possible long-term Companion is to be. Even the need for 
politeness, which is widely emphasized (e.g. Wallis et ah, 2001), especially in the form of 
responses to “face saving” and “face attacks” (Brown and Levinson, 1987) is not universally 
accepted. Cheepen and Monaghan (1997) showed that, at least in things like automated teller 
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machines (ATMs), users were resistant to notions of politeness that they found excessive, 
especially coming from machines who could not “really” be polite, or indeed anything. Along 
the latter lines, Moore (2003) has argued that Companions should not be too realistic, too 
human-like in their avatars, and not because of any “uncanny valley effect” (Mori, 1970) that 
induces repulsion, but because such realism in avatars, or in the quality of artificial voices, 
raises expectations too high, in terms of understanding what is said, which in turn leads to 
disillusion. This view argues for more abstract or cartoon-like Companion avatars, from 
whom any apparent understanding might be received with gratitude. 


David Levy (in this volume) draws attention to the extraordinary Tamagotchi phenomenon: 
of intelligent people making an effort to care for, and cosset, primitive machines without any 
language at all. At a similar level of machines, the classic work of Reeves and Nass (1996) 
similarly drew attention to an apparent desire of users not to “offend” office machines they 
dealt with on a daily basis. Nonetheless, there is a great deal of counter-evidence (Wallis, 
ibid.) that users, knowing a machine to be one, will take the opportunity to abuse and insult it 
immediately, often, and without cause. 


Taken with Bryson’s (this volume) arguments for keeping machines in a “slave” posture with 
no attributions of sentience or rights, this range of evidence suggests there is not yet a 
consensus on how to view or deal with machines, nor agreement on what social and personal 
role we want them to adopt with us. It may well turn out that there is no single level of 
“posture” of Companions towards us, but rather a whole range of possibilities depending on 
the function the Companion is to perform and the personal preferences of a user for a given 
Companion character type, a preference that might change with time. 
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Nonetheless, I personally find the lady’s Companion list above an attractive one: it eschews 
emotion beyond the linguistic, it implies care for the mental and emotional state of the user, 
and I would personally find it hard to abuse any computer with the characteristics listed 
above. It is no accident, of course, that this list fits rather well with the aims of the Senior 
Companion demonstrator in the Companions project*. But the project has begun 
simultaneously with a Health and Fitness Companion for the more active, one sharing much 
of its architecture with the first, and one that would require something in addition to the list 
above: the “personal trainer” element of weaning, coaxing and threatening which adds 
something quite different to that list, and something very close to the economic-game bargain 
of the kind discussed in some detail by Will Lowe in his chapter. 


In conclusion, it may be worth making a small clarification about the word “avatar” that 
sometimes dogs discussions in these areas: those working in computing the human-machine 
interface often use the word to mean any screen fonn, usually two-dimensional, that 
simulates a human being, but not any particular human being. On the other hand, in the 
virtual reality and game worlds, such as Second Life (URL below) or the “mii” avatars of the 
wii, an avatar is a manifestation of a particular human being, an alternative identity that may 
or may not be similar to the owner in age, sex, appearance etc. These are importantly 
different notions and confusion can arise when they are conflated or confused: in current 
Companions project demonstrations, for example, a number of avatars in the first sense are 
used to present the Companion’s conversation on a computer or mobile phone screen. 
However, in the case of a long-tenn computer Companion that could elicit, through 
prolonged reminiscence, details of its owner’s life and perhaps train its own voice in 
imitation, one might go further, since research shows that more successful computer 
conversationalists are as like their owners as possible. One could then approach the point 
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where a Companion could approximate to the second sense of “avatar” above, namely an 
avatar of its owner, which it would progressively resemble, as dogs are said to do. 


Such situations could lead to what are, at the moment, wildly speculative ones: that of a 
Companion acting as its owner’s agent, on the phone or world wide web, perhaps holding 
power of attorney in case of an owner’s incapacity and, with the owner’s advance permission, 
perhaps even being a source of conversational comfort for relatives after the owner’s death. 
All these situations are at present absurd, but perhaps we should be ready for them. 


In summary: in this chapter 
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Section Four - Design Issues: Building a Companion 
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The use of affective and attentive cues in an empathic computer-based 
Companions - Nikolaus Bee, Elisabeth Andre, Thurid Vogt and Patrick Gebhard 


Abstract 


Recently, a number of research projects have been started to create virtual agents that do not 
just serve as assistants to which tasks may be delegated, but that may even take on the role of 
a Companion. Such agents require a great deal of social intelligence, such as the ability to 
detect the user’s affective state and to respond to it in an empathic manner. The objective of 
our work is to create an empathetic listener that is capable to react on affective and attentive 
input cues of the user. In particular, we discuss various forms of empathy and how they may 
be realized based on these cues. 


Introduction 


Listening is a component of any dialogue, and to make your counterpart feel understood and 
cared for, you need the capabilities of an empathetic listener that appears attentive and 
sensitive towards the emotional state of others. Consider, for example, a situation where your 
best friend fails an exam that was very important for him as he decides about graduating. 
When your friend tells you about his situation, you may show different kinds of empathetic 
reaction. In the easiest case, you show an emotional expression that is similar to that of your 
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friend. Your friend will usually interpret your behavior as a sign that you share his feelings 
and might finally feel better, as in the motto: a problem shared is a problem halved. Instead of 
simply mirroring your friend’s emotional state, you may also show an emotional state as 
response to your friend’s emotional state that is related to that of your friend, but not 
necessarily the same. For example, if your friend looks sad, you might show him by a 
compassionate facial expression that you are sorry about his sadness. Responding to 
somebody’s emotional state by conveying a so-called good-will emotion, such as sorry-for or 
happy-for, is called reactive empathy. Another possibility is trying to understand the current 
situation and thinking about possibilities for making your friend feel better. For example, you 
might encourage your friend by telling him that you also failed some exams but there is 
always a second chance. We call this fonn of empathy deliberate empathy. Deliberate 
empathy requires a deeper level of affective mind reading than parallel or reactive empathy. 
In order to comfort your friend, you need to be able to reason about his affective appraisal 
processes. In Cognitive Science, this ability is called an affective theory of mind. 


Recently, a number of research projects have been started to create virtual agents: Such 
agents require a great deal of social intelligence, such as the ability to detect the user’s 
affective state and to respond to it in an empathic manner. While a lot of work has been done 
on the creation of affective output for virtual characters, less work was done on the smooth 
integration of recognition and display processes. The objective of our work is to create an 
empathetic listener that is capable to react on affective and attentive input cues of the user. 
Such an agent has to appear sensitive by appropriately responding to the user’s affective 
state. Furthermore, it has to appear attentive by establishing frequent eye contact with the 
user without being obtrusive. 
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Related Work 


Prendinger and colleagues (Prendinger, Dohi, Wang, Mayer, & Ishizuka, 2004) developed an 
empathic Companion that accompanies a user in a virtual job interview. This system 
measures the users’ physiological state (skin conductance and electromyography) in real-time 
and interprets it as emotion. The virtual agent reacts with empathic feedback on the users’ 
current affective state. The reaction is calculated using a Bayesian net, which takes the 
physiological state and the user’s answers to the interview into account. Gratch et al. (Gratch, 
Wang, Gerten, Fast, & Duffy, 2007) describe a system to establish rapport in human-machine 
dialogs. They detect speech and head orientation from the user to create a responsive 
dialogue agent. Speech detection is used for detecting backchannel opportunity points, 
disfluencies, questions and loudness and the head tracker detects head nods, shakes, gaze 
shifts and posture shifts. Their system is mainly meant to create contingent nonverbal 
behavior and neither takes the user’s emotional state into account nor intends to react with 
affective behavior. McQuiggan et al. (McQuiggan, Robison, Phillips, & Lester, 2008) created 
a system for empathic virtual agents. Using a machine learning approach, they automatically 
detennine appropriate parallel and reactive empathy behaviors. The system is trained by the 
users’ in-game behavior. It does not consider the real users’ actual emotional state. Ochs et al. 
(Ochs, Pelachaud, & Sadek, 2008) created an empathic model for virtual agents by 
combining a corpus-based approach with theoretical work on emotions. Their model was 
applied to an agent that provides users with information on their email. The system infers the 
user’s emotional state based on an analysis of the dialogue and computes an empathetic 
reaction to it. Boukricha (Boukricha, 2008) proposes a computational model for empathic 
virtual agents that distinguish between sharing the emotions of others (mimicry) and 
understanding the emotions of others cognitively (role-taking). Detection of the users’ current 
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emotional state, which will be mapped to a virtual character, is planned by using facial 
feature detection. There is an ongoing project named SEMAINE (Schroder, Cowie, Heylen, 
Pantic, Pelachaud, & Schuller, 2008) that deals with building Sensitive Artificial Listeners, 
which are designed to sustain the communication with a person. Their system will recognize 
and generate non-verbal behavior in real-time. 


Example 


Our objective is the creation of an empathic listening agent that analyzes the user’s emotive 
state and responds to it in real-time. To realize such an agent, we have been experimenting 
with several metaphors, such as that of a virtual pet or that of a virtual butler. While the 
virtual pet is not able to verbally respond to the user’s state, the virtual butler provides both 
verbal as well as non-verbal feedback. Imagine the user has a rather bad day and is talking to 
the agent with a gloomy voice. In the case of parallel empathy, the agent would simply mimic 
the user’s emotive state and show gloom as well (see Figure 1 middle). In contrast to that, 
reactive empathy requires the agent to decide which emotion to display as a response to the 
user’s emotive state. Here, the agent’s emotion does not necessarily coincide with the user’s 
emotion. The agent might, for example, decide to show “pity for” (see Figure 1 right) the 
human for example suggested by (Pelachaud & Poggi, 2002) with the head aside and the 
inner eyebrow raised or to cheer the user up by showing encouragement (see Figure 1 left). In 
addition, the agent might provide simple verbal feedback, such as “I am feeling bad as well.” 
(in the case of parallel empathy) or “I am very sorry you are feeling so bad.” (in the case of 
reactive empathy). 
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Figure 1: The virtual character Alfred is designed utilizing FACS to compose facial 
expressions (left: joyful, middle: gloomy and right: pity-for). 


Affective Interaction 


Our architecture provides components to sense affective and attentive states in the user, a 
component to process affective states for parallel and reactive empathy and a component to 
display affect and attention with virtual characters. Currently, emotion recognition is based 
solely on an analysis of acoustics using a microphone. The user’s focus of attention is tracked 
employing an eye tracker. Further, we use a realistic virtual character with highly expressive 
facial emotions to display appropriate facial expressions. 


A lot of effort has been spent on the implementation of emotional models. Prominent 
examples include EMA (Gratch & Marsella, 2004) and ALMA (Gebhard, 2005). We use 
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ALMA because it includes a language for defining emotion display appraisal rules to 
describe how a virtual agent appraises its own emotional displays and those of other agents. 
These rules are useful when implementing an empathetic listener that responds to the user’s 
emotional state. In addition, ALMA combines an appraisal mechanism with a dimensional 
representation of emotions which is of benefit when describing how emotions evolve over 
time. We combine the component for affective sensory input with an emotional model for our 
empathic listener that allows it to act or react to the users’ feelings. 


Affect and Attention Sensing 


For sensing affect from users’ voice, we use EmoVoice (Vogt, Andre, & Bee, 2008). It is a 
framework that provides support for the acquisition of emotional speech corpora and the 
training of classifiers. Furthennore, it is suitable both for offline as well as online vocal 
emotion recognition. The framework is intended to be used by non-experts and therefore 
comes with an interface to create an own personal or application specific emotion recognizer. 
Table 1 shows some examples for recognition accuracies achieved by Emo Voice on different 
corpora. The FAU Aibo Emotion Corpus (Steidl, 2009) is a spontaneous emotion corpus of 
children interacting by voice commands with an Aibo pet robot. The Berlin database 
(Burkhardt, Paeschke, Rolfes, Sendlmeier, & and Weiss, 2005) is a corpus of acted emotional 
speech. Furthennore, we had students record their own corpora in order to test how easy the 
process of classifier building and corpus creation is for people who are unfamiliar with the 
task. 
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Table 1: Examples for recognition accuracies achieved by EmoVoice 


Corpus 

Number of emotion 

classes 

Speaker 

independent? 

Recognition accuracy 

Aibo 

4 

Yes 

57% 

Berlin 

7 

Yes 

83 % 

Students 

4 

No 

24-74 % 


4 

Yes 

41 % 


2 

Yes 

60-70 % 


Eye gaze may be considered one of the most important means to indicate attention in a 
dialogue. Kleinke shows correlations between attention and gaze (Kleinke, 1986). For 
instance, interviewers are found more attentive when their gaze is relatively high. 
Furthermore, interviewees give shorter responses when an interviewer is not looking at them. 
To measure the amount and orientation of the user’s gaze, we connect the eye tracker with 
the virtual world. Ray casting allows us to map the screen coordinates obtained from the eye 
tracker to the objects in the virtual world. In this way, we are able to detect whether the user 
looks at the virtual agent, the left eye or the right eye or something else in the virtual scene. 


Affect Model 


For the affect simulation in real-time, we rely on ALMA Bio and extended version of the 
computational model ALMA (Gebhard, 2005). ALMA Bio allows the processing of values 
directly derived from sensor signals (e.g. speech or bio signals). In this context the signals are 
treated as unspecific emotions that contain individual (measured) pleasure, arousal and 
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dominance (PAD, see below) values. These measured emotions are used as input to change or 
intensify the current mood. 


ALMA provides three affect types as they occur in human beings: (1) emotions reflect short¬ 
term affect that decays after a short period of time; (2) moods reflect medium-term affect, 
which is generally not related to a concrete event, action or object; and (3) personality reflects 
individual differences in mental characteristics and affective dispositions. 


ALMA implements the cognitive model of emotions developed by Ortony, Clore and Collins 
(OCC) (Ortony, Clore, & Collins, 1988) combined with the “Big Five” model of personality 
(McCrae & John, 1992) and a simulation of mood based on the PAD model (Mehrabian, 
1995). The relations between the different affect types are a central part of the affect 
simulation: 


• A given personality defines a default mood and influences the intensities of different 
emotions. 

• The current mood amplifies or dampens the intensities of emotions. 

• Emotions as short term affective events influence the longer-tenn mood. 


Elicited emotions influence an individual’s mood. The higher the intensity of an emotion is, 
the greater the particular mood changes. Emotions usually do not last forever. Over a specific 
period the intensity of emotions decays and the influence on the current mood fades. 
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The current mood also influences the intensity of emotions (Neumann, Seibt, & Strack, 
2001). This determines, for example, the intensity increase of joy and the intensity decrease 
of distress, when an individual is in an exuberant mood. Mood is represented by a triple of 
the mood traits pleasure (P), arousal (A) and dominance (D). The mood’s trait values define 
the mood class. If, for example, every trait value is positive (+P, +A, +D), the mood is 
exuberant. 


Alfred - Affect Display 


The artificial head, Alfred is capable of generating empathy through displaying affective 
facial expressions and generating attentive gaze behavior. While (Niewiadomski, Ochs, & 
Pelachaud, 2008) concentrate on improving empathic expressions by generating complex 
facial expressions like superposition and masking, we focus on generating empathic feedback 
by combining affective facial expressions and attentive gaze behavior that reacts on the user’s 
eye gaze. Composing facial expressions with the FACS-based approach, gives Alfred a great 
flexibility to generate a large number of affective expressions. Three examples of affective 
facial expressions can be seen in Figure 1. The attentive gaze behavior can be varied to 
generate attention dependent on the user’s state. In addition, Alfred is able to speak and to 
nod his head. Alfred is able to talk with a text-to-speech (TTS) system or by playing pre¬ 
recorded (emotional) audio files with synchronized lip movements. Although these 
capabilities are not yet used as empathic feedback cues, one can easily imagine how Alfred’s 
capabilities can be enhanced to head nodding or verbal feedback such as “Mm Hm” or “Uh 
huh” while showing empathy. 
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Facial Expression 

Ekman and Friesen developed the Facial Action Coding System (FACS) to classify human 
facial expressions (Ekman & Friesen, 1978). FACS divides the face into action units (AFT) to 
describe the different expressions a face can display (e.g. inner brow raiser, nose wrinkle, or 
cheek puffer). Although FACS was originally designed to analyze natural facial expressions, 
it turned out to be usable as a standard for production purposes, too. That is why FACS based 
coding systems are used for the generation of facial expressions displayed by virtual 
characters, like Kong in Peter Jackson’s King Kong (Sagar, 2006). But the usage of FACS is 
not only limited to virtual characters in movies. The gaming industry (with Half-Fife 2 by 
Valve), also utilizes the FACS system to produce the facial expressions of their characters 
(Valve). 


Alfred is a butler-like character, and uses these action units to synthesize an unlimited set of 
different facial expressions. The action units were designed using morph targets and thus give 
the designer power to define the facial expression outlook (Bee, Falk, & Andre, 2009). 


We chose the FACS-based approach for our facial animation system, because of the Facial 
Expression Repertoire (FER), which maps over 150 emotional expressions to the action units 
of FACS. Not only does it explain in detail, which action unit must be activated for certain 
facial expressions, it provides a rich dataset of videos which show how the action units ought 
to be designed. 
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Alfred’s mesh has a resolution of about 21.000 triangles. For displaying more detailed 
wri nk les in the face, nonnal maps baked from a high-resolution mesh are used (Oat, 2007). 
The morph targets for the action units are modelled using the actor’s templates from the FER. 
For rendering the character and its animations the Horde3D Game Engine (Horde3D, 2008) 
is used. 


Gaze Behavior 

Fukayama et al. propose a gaze behavior model for virtual characters based on amount and 
mean duration of gaze and averted gaze orientation (Fukayama, Ohno, Mukawa, Sawaki, & 
Hagita, 2002). They rated two groups of attributes: One was named “friendliness” and was 
correlated in their study with attributes such as friendly, wann, sociable, tolerant, flexible, 
attentive and coordinative. The other was named “dominance” and correlated with assured, 
strong, successful, responsible and careful. Fukayama and colleagues found that a medium 
amount of gaze, a mean duration between 500 to 1000 ms conveys a “friendly” gaze 
behavior. The orientation of the gaze direction did not play a major role in distinguishing 
between the friendly and dominant gaze behavior, except that a downward gaze was 
considered as less dominant. 


Alfred’s gaze behavior was extended with further parameters as our virtual agent is capable 
of reacting to the user’s current focus of attention. A confidence value defines the agent's 
level of self assurance and influences the probability that the agent initiates the gaze 
interaction. The maximal and minimal duration of mutual gaze can be set as well. 
Furthermore, we may indicate the maximal duration the virtual agent can gaze around. 
Finally, we may specify how long the virtual agent waits until the user responds with mutual 
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gaze. These parameters are stored in a XML file and can be easily adjusted. The virtual agent 
is able to direct his gaze using his eyes only or his head and his eyes in combination. 


Speech 

The system interfaces the Microsoft Speech API to synchronize the audio output with the lip 
movements. This allows us to use any text-to-speech system that supports SAPI 5. As the 
quality of common TTS systems may not be satisfactory, we integrated a module to 
synchronize pre-recorded audio speech files with the lip movements of the virtual character. 
This allows us to use highly emotional sentences or affect bursts to be spoken through a 
virtual character. As FACS defines several action units involving mouth muscles (e.g. lip 
funnelled, lip tightener, mouth stretch), we utilize the FACS system for lip movements. The 
approach is similar to displaying facial expressions. The output from the editor to modify the 
single action units is stored in a XML file. Reusing the FACS approach for visemes enables 
Alfred to display facial expressions and lip movements parallel. 


Empathetic feedback 


For the modelling of empathic feedback, we employ ALMA to simulate the agent’s and the 
user’s emotional state. Both states are independently represented. From the point of view of 
the agent, the user’s emotional state is simulated according to the agent’s own model of 
emotions (ALMA) but based on appraisal rules that the agent assumes the user has. Due to 
the fact the personality of the user is not known, ALMA is used with neutral personality 
values. This allows a simulation of mood tendencies, which is depicted in Figure 2. 
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Figure 2: Screenshots of mood tendency simulation with ALMA. 

Figure 2 shows the mood simulation based on neutral personality settings (centre). Each 
elicited emotion impacts the neutral mood. The mood is altered according to an elicited 
emotion’s relation to the PAD mood trait values (e.g. Joy is related to +P, +A, +D values, 
right side. Distress is related to -P, +A, +D values, left side). The simulated mood represents 
a general mood tendency that is influenced by emotions. If no emotion is elicited, the mood 
slowly decays to the neutral mood. 

Our approach to the modelling of empathic feedback consists of 3 stages: 

1. Parallel Empathy (Mimicry of the user’s emotion) 

2. Reactive Empathy (Reacting to the user’s emotion) 

3. Deliberate Empathy (Altering the user’s emotion) 


In the case of parallel empathy and reactive empathy, emotional responses are generated 
using the agent’s current mood PAD values (representing the agent’s emotional state) in 
order to influence the agent’s facial expressions. In the case of parallel empathy, the PAD 
values representing the agent’s emotional state correspond to the PAD values representing the 
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user’s emotional state. In the case of reactive empathy, PAD values for the agent’s emotional 
state are assigned based on an appraisal mechanism. 


While this approach is suitable for parallel empathy and less complex forms of reactive 
empathy, such as good-will emotions, the handling of communicative empathy that is shown 
with the aim to change the user’s emotional state requires further work. 


Parallel Empathy 


In the case of parallel empathy, the agent mimics the affective state of the user. Hoffman 
defines mimicry as primitive, automatic and involuntary which is a process of the two steps 
“imitation” and “feedback”. Humans automatically imitate and synchronize changes in their 
facial expression and voice. Through changes in the musculature, these synchronizations 
produce feelings that match (Hoffman, 2000). 


We used ALMA to create a model of the user’s emotional behavior. In the case of mimicry, 
the user’s affective input is directly mapped onto PAD values. Currently, we just rely on 
vocal emotion recognition for the mapping even though the user’s gaze conveys information 
on his or her emotional state as well: E.g the PAD values for dominance could be derived 
from gaze. To avoid jumps between emotional states, we smooth the input signals during 
affective interpretation. We make use of a decay factor to account for the fact that the 
importance of a signal decreases over time (See Gilroy, et ah, 2008). The emotional state of 
the user in the model is taken as a basis for the character’s emotional behavior which just 
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mirrors the user’s emotional state. The user’s emotional state is represented by PAD values 
that were derived from vocal emotion recognition. The same PAD values may be used to 
generate facial expressions with Alfred. 


Reactive Empathy 

For reactive empathy, we make use of ALMA’s Emotion Display Appraisal Rules that 
describe how a character appraises the emotion displays of others. Those rules basically map 
an observed emotion display onto Appraisal Tags, which are shortcuts to the OCC emotion 
eliciting conditions. ALMA computes emotions based on those tags, and provides these 
Appraisal Tags classes: 12 Event Appraisal Tags (e.g. GoodEvent, BadEvent, 
BadEventForGoodOther, GoodLikelyFutureEvent ...), 4 Action Appraisal Tags (e.g. 
GoodActSelf, BadActSelf, GoodActOther, BadActOther), and 2 Object Appraisal Tags (e.g. 
NiceThing). For reactive empathy, the agent assumes that the user is likeable, which has an 
impact on how the agent appraises a user’s emotional display. Figure 3 shows part of the 
rules used. 


Disappointment, Distress, Remorse, SorryFor -> BadEventForGoodOther 1.0 
Joy, Gratification, HappyFor, Satisfaction -> GoodEventForGoodOther 1.0 


Figure 3. Example appraisal rules for emotional displays. 


For events that trigger an emotional response in a user, our agent is capable of the following 
reactive empathy emotions and show appropriate reactions: HappyFor, SorryFor, 
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Resentment, and Gloating. This depends basically on the agent liking the user. For actions, 
our agent is capable of showing appropriate reactions for the emotions Shame, Reproach and 
for the compound emotion Anger. For objects, Alfred is able to show an appropriate reaction 
to the emotion Dislike due to the fact it can be easily observed (e.g. a face showing 
expressing disgust). 


Deliberate Empathy 


At a higher level of reactive empathy, the agent does not display emotions it feels, but tries to 
express emotions that might help to put the user into a better emotional state. Reactive 
empathy requires the agent to put himself into the place of the user in order to decide on an 
appropriate emotional display. For example, if the user is afraid of failing in an exam, the 
agent might tell the user that he will manage because of his good revision. The agent praises 
the user which is then perceived by the user a positive event. In addition, the agent utters a 
statement which decreases the likelihood of the negative event (namely the user, failing in the 
exam). In the deliberate mode, it is crucial for the virtual agents to be able to express 
themselves verbally - specifically if their emotional display does not coincide with the user’s 
emotion. For example, if the user is sad and the agent is smiling, the user might not perceive 
this as encouragement, but as gloating (see Figure 1 middle). ALMA can potentially serve as 
an affect model for a higher level of reactive empathy. It could be used to realize an affective 
theory of mind by running a simulation on a planned utterance. These appraisal methods 
directly support this approach (e.g. OCC appraisal variable-value pairs, Appraisal Tags, 
Emotion Display Tags, Dialog Act Tags, and direct PAD input). Suppose the agent intends to 
generate an utterance, such as “Don’t worry about the exam. You are so well prepared.” The 
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good preparation of the user decreases the likelihood of the negative event (Fail Exam). 
According to the OCC model, the intensity of the emotion Fear would be decreased. 


Conclusion 


Our approach of creating an affective and attentive virtual listener that reacts in an empathic 
manner to the user’s input focuses on the interpretation and generation of affective and 
attentive cues. We have shown how an emotional model can be used to simulate different 
levels of empathy. Parallel empathy can be handled by mapping input signals onto PAD 
values which are then transfonned into appropriate output signals. In contrast, reactive 
emotions, such as sorry-for or happy-for, require an appraisal mechanism that detennines 
how the agent responds to the display of other agents. Even if the agent does not understand 
what the user is saying, he might still be able to communicate via affective cues that he is 
caring for the user simply by emotionally responding to the user’s emotions. A great 
challenge represents deliberate empathy that aims at changing the counterpart’s emotional 
state. In this case, the agent usually needs to understand why the user is feeling a certain 
emotion in order to give appropriate feedback. 

Another non-trivial issue is to decide which fonn of empathy a virtual agent should show. 
One criterion to consider might be the personality the agent is supposed to portray. For 
example, an agreeable agent tends to show parallel empathy more frequently than a 
disagreeable agent. Our system provides the possibility of testing different behavior variants 
in order to figure out under which conditions which fonn of empathy is more appropriate. 
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GRETA: Towards an Interactive conversational virtual companion 

Elisabetta Bevacqua, Ken Prepin, Radoslaw Niewiadomski, Etienne de Sevin 
and Catherine Pelachaud 


Abstract 

In this chapter we present our work toward building a conversational Companion. Conversing 
with partner(s) means to being able to express one’s mental and emotional state, to be a 
speaker or a listener. One needs also to adapt to ones partner’s reactions to what one is 
saying. We have developed an interactive ECA platform, Greta (Pelachaud, 2005). It is a 3D 
virtual agent capable of communicating expressive verbal and nonverbal behaviors as well as 
listening. It can use its gaze, facial expressions and gestures to convey a meaning, an attitude 
or an emotion. Multimodal behaviors are tightly tied with each other. A synchronization 
scheme has been elaborated allowing the agent to display a raised eyebrow or a beat gesture 
on a given word. According to its emotional or mental state, the agent may vary the quality of 
its behaviors: it may use a more or less extended gesture, the arms can move at different 
speeds and with different accelerations (Mancini & Pelachaud, 2008). The agent can also 
display listener behavior (Bevacqua et al, 2008). It interacts actively with users and/or other 
agents providing appropriate timed backchannels. Interaction also means the interactants 
ought to adapt to each others’ behaviors and dynamic coupling between them needs to be 
considered (Prepin & Revel, 2007). 


Introduction 
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Research has shown that people tend to interact with computers characterized by human-like 
attributes as if they were really humans (Nass et ah, 1997, Reeves and Nass, 1996). For 
example, in their studies Nass and Reeves saw that, when interacting with computers, people 
apply rules of politeness and felt uneasy when large faces were displayed on a screen, as if 
the talking head was invading their personal space (Reeves and Nass, 1996). Consequently, 
human-machine interface designers should aim to implement interactive systems that 
simulate human-like interaction. The more this type of interface is consistent with a human 
style of communication, the more their use will become easy and accessible (Ball and Breese, 
2000). Such a level of consistency could be reached using humanoid artefacts able to apply 
the rich style of communication that characterizes human conversation. Recent technological 
progress has made the creation of this type of humanoid interfaces, called Embodied 
Conversational Agents (EGAs), possible. An ECA is a computer-generated animated 
character that is able to carry on natural, human-like communication with users (Cassell et 
ah, 2000b). For this purpose, the researchers engaged in ECA development share common 
goals: They want to implement agents which can simulate humans' verbal and non-verbal 
behavior, like speaking in natural language, making gestures, displaying facial expressions, 
shifting their gaze and moving their head like humans do in everyday life. Researchers also 
aim to provide these virtual artefacts with the capability of understanding what humans say, 
interpreting their non-verbal signals and using all this information to decide how to react and 
respond. 

Several EGAs have been developed so far. They exhibit human-like communicative 
capabilities: they can talk, listen, grab one’s attention, look at another one, show emotion, and 
so on (Cassell et ah, 2000a; Gustafson et ah, 1999; Gratch and Marsella, 2004; Kopp and 
Wachsmuth, 2004; Pelachaud, 2005; Heylen, 2006; Gratch et ah, 2007). They can play 
different roles, from a Companion for old people or young children, to a virtual trainee, a 
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game character, a pedagogical agent or even a web agent (Johnson et ah, 2005; Moreno, in 
press; Cassell et ah, 1999; Hall et ah, 2006; Bickmore et al, 2007). 


In the remainder of this chapter we describe the implementation of our ECA system and 
describe in more details the modules linked to listener model. In the next section we 
introduce the SAIBA framework that Greta complies with the representation languages used 
to control the agent are presented. Then each module of the ECA system is explained, and the 
chapter ends with a description of applications of our system. 


GRETA 


GRETA’s architecture follows the design methodology proposed in (Thorisson et al., 2005) 
and is compatible with the SAIBA framework (Vilhjalmsson et al., 2007). Its architecture is 
modular and distributed, and each module exchanges information and data through a central 
message system by means of whiteboards as defined by Thorisson (Thorisson et al., 2005). It 
allows internal modules and external software to be integrated easily. The system is designed 
to be used in interactive applications working in real-time. Interactive applications of our 
system were developed within the EU-Projects eNTERFACE 53 , SEMAINE" 4 and 
CALLAS 55 . 
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The SAIBA framework 


SAIBA 56 is an international research initiative whose main aim is to define a standard 
framework (i.e. a conceptual architecture and associated standard languages) for the 
generation of virtual agent behavior (Vilhjalmsson et ah, 2007) (see Figure 1). 


feedback 



feedback 


Figure 1: SAIBA architecture. 


It defines a number of levels of abstraction, from the computation of the agent's 
communicative intention, to behavior planning and realisation. The Intent Planner module 
decides the agent's current goals, emotional state and beliefs, and encodes them into the 
Function Markup Language (FML) (Heylen et ah, 2008), a standard language still being 
defined. To convey the agent's communicative intentions, the Behavior Planner module 
schedules a number of communicative signals (e.g., speech, facial expressions, gestures) 
which are encoded with the Behavior Markup Language (BML). The BML specifies the 
verbal and nonverbal behaviors of ECAs (Vilhjalmsson et ah, 2007) Finally the task of the 
third element of the SAIBA framework, Behavior Realizer, is to realize the behaviors 
scheduled by the Behavior Planner. It receives input in the BML format and it generates the 
animation. 
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SAIBA examples 


There exists already several implementations like SmartBody (Thiebaux et al., 2008) and 
BML Realizer (Arnason and Porsteinsson) that are SAIBA compatible. 

SmartBody (Thiebaux et ah, 2008) is a modular, distributed open-source framework for 
animating EGAs in real time. It corresponds to the Behavior Realizer module of the SAIBA 
architecture. It takes as input BML code (including speech timing data and the world status 
updates); it composes multiple behaviors and generates character animation synchronized 
with audio. The verbal content is generated by an external TTS system. BML used within 
SmartBody is a subset of the standard version (Thiebaux et al., 2008); but it offers some 
extensions as well. SmartBody can be used with the Nonverbal Behavior Generator (Lee and 
Marsella, 2006) that corresponds to the Behavior Planner in the SAIBA framework. It is a 
rule-based module that generates BML annotations for nonverbal behaviors from the 
communicative intent and speech text. On the other hand, SmartBody can be used with 
different characters, skeletons and even different rendering engines. 


BMLRealizer created in the CADIA lab is another implementation of the Behavior Realizer 
layer of the SAIBA framework. It is an open source animation toolkit for visualizing virtual 
characters in 3D environment that is partially based on the SmartBody framework. As input it 
also uses BML; the output is generated with the use of the Panda3D rendering engine. 
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GRETA’S implementation of SAIBA 


GRETA’s architecture is an almost full implementation of the SAIBA framework. It is 
composed of three main modules (see Figure 2), offering solutions for the Behavior Planner 
and the Behavior Realizer and a partial implementation of the Intent Planner. 

In the SAIBA standard the Intent Planner is dedicated to generate intentions of a Virtual 
Speaker agent. To be able to control a Listener agent, we have introduced the Listener Intent 
Planner, which generates automatically the communicative intentions of the listener. In the 
current state of our system, when the agent is the Speaker (and not the Listener) its intentions 
are prc-dclined manually in an FML-APML input file (FML- APML is a follower of the 
language APML (DeCarolis et al, 2001)). In future works, they should be generated by a 
Speaker Intent Planner. 


2: GRETA's architecture. 


Figure 
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Each communicative intention generated by the Intent Planner (both when the agent is 
Speaker or Listener) is transmitted to the Behavior Planner, in FML-APML language. The 
Behavior Planner proposes a list of corresponding possible nonverbal behaviors, written in 
BML language. These behavior signals are sent to the Behavior Realizer that generates 
MPEG-4 Facial Animation Parameters (FAP) and Body Animation Parameters (BAP) 
frames. Finally, the animation is played in the FAP-BAP Player. All modules are 
synchronized by the Central Clock and communicate with each other through the Psyclone 
whiteboard (Thorisson et al., 2005). 


Intent Planner 


The Intent Planner module has the task of computing the agent's intentions when a listener or 
a speaker, and consists of two modules: the Listener Intent Planner and the Speaker Intent 
Planner that calculate the agent's behavior. So far just the module to compute the listener's 
intentions has been implemented and the Speaker Intent Planner is still under construction. 
Together, these two modules will correspond to the Intent Planner in the SAIBA framework. 


Listener Intent Planner 


The Listener Intent Planner computes the backchannel signals that the agent provides while 
listening. This module implements three type of backchannels: reactive, response and 
mimicry. Reactive backchannels derive from a first process of perception of the speaker's 
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speech and they show contact and perception; response backchannels are generated by a more 
aware evaluation that comprehends memory and cognitive process (Kopp et ah, 2008). 
Finally, the mimicry backchannels derive from the imitation of the speaker's behavior. By 
mimicry we mean the behavior displayed by an individual who does what another person 
does (Van Baaren, 2003). This type of behavior has been proven to play quite an important 
and positive role during conversations (Warner et ah, 1987; Chartrand and Bargh, 1999). 


Several researchers (Allwood et al., 1992; Poggi, 2005) have noticed that listeners can emit 
signals with different levels of intentionality: they react instinctively to the speaker’s 
behavior, generating signals at a very low level of control; or they can decide consciously to 
emit a signal in order to show their reaction to the speaker’s speech and even act with the 
intent to influence the speaker's behavior. The Listener Intent Planner generates listener's 
signals with a low level of awareness. We do not take into account the agent's intention to 
provide a backchannel signal; backchannels are emitted unintentionally. 


To generate backchannel signals, the Listener Intent Planner uses two modules called 
response/reactive backchannel and mimicry. In order to decide when a backchannel should be 
emitted and to select which communicative functions the agent should transmit, the Listener 
Intent Planner component needs three data as input: 

1. the user’s verbal and nonverbal behavior that is tracked through a video camera and a 
microphone; 

2. the user's estimated interest level. Such a level is calculated by evaluating the user's 
gaze, head and torso direction within a temporal window (Peters et al., 2005); 
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3. the agent's mental state directed towards the interaction. 


Research has shown that there is a strong correlation between backchannel signals and the 
verbal and nonverbal behaviors performed by the speaker (Maatman et ah, 2005; Ward and 
Tsukahara, 2000). Models have been elaborated that predict when a backchannel signal can 
be triggered based on a statistical analysis of the speaker's behaviors (Maatman et ah, 2005; 
Morency et ah, 2008; Ward and Tsukahara, 2000). 


From the literature (Maatman et ah, 2005; Ward and Tsukahara, 2000) we have fixed some 
probabilistic rules to prompt a backchannel. Our system analyses speaker's behaviors looking 
for those that could prompt an agent's signal; for example, a head nod or a variation in the 
pitch of the user's voice will trigger a backchannel with a certain probability. When a rule is 
found, the probability that certain behavior provokes a backchannel depends also on the 
user's estimated level of interest. This value is used by the system to vary the backchannel 
emission frequency: when the interest level decreases it may be a sign that the user might 
want to stop the conversation (Schegloff and Sacks, 1973), consequently the agent shows 
disengagement as well and provides less and less backchannels. 


When a backchannel is triggered, the response/reactive backchannel, and mimicry modules 
compute which type of backchannel should be displayed. The response/reactive backchannel 
module uses infonnation about the agent's beliefs towards the speaker's speech to calculate 
the backchannel signal. This infonnation is stored in the agent's mental state representation 
that describes how the agent is reacting to the user's speech. In our system, the agent's mental 
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state is represented as a list of the listener's communicative functions. We use Allwood's and 
Poggi's taxonomies of communicative functions of backchannels (Allwood et al., 1992; 
Poggi, 2005): understanding and attitudinal reactions (liking, accepting, agreeing, believing, 
being interested). The response/reactive module takes into account the agent's mental state to 
decide which communicative functions the agent should convey. Then, the appropriate 
signals to display will be selected from a backchannel lexicon that we have elaborated in 
previous studies (Bevacqua et al., 2007; Heylen et al., 2007). When no information about the 
agent's beliefs towards the speaker's speech is given, the response/reactive module selects a 
pre-defincd backchannel from among those signals that have been proven to show contact 
and perception, like head-nod and raise-eyebrows. 


When fully engaged in an interaction, mimicry of behaviors between interactants may happen 
(Lakin et al., 2003). The mimicry module detennines which signals would mimic the agent. 
So far we are considering solely speaker's head movements as the signals to mimic. A 
selection algorithm (explained below) detennines which backchannel to display among all 
the potential signals that are outputted by the two modules. 


Behavior Planner 


The Behavior Planner takes as input both the agent's communicative intentions specified by 
the FML-APML language and some agent characteristics. The main task of this component is 
to select, for each communicative intention, the adequate set of behaviors to display. The 
output of the Behavior Planner is described in the BML language. It contains the sequence of 
behaviors with their timing infonnation to be displayed by our virtual agent. 
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Reactive Behavior 


The mutual adaptation necessary to enable verbal interaction between an ECA and a human 
is, in some ways, highly cognitive: the speaker can have to re-plan its speech depending on 
listener’s reactions; the emotions of the agents (speaker or listener) can change throughout 
the dialogue and influence its behavior. 


However this mutual adaptation is also, in other ways, mostly reactive: it is not linked to the 
meaning of words or to changes within the mental state of the agent, but more to 
psychophysical and dynamical properties of the exchanged signals (Murray and Trevarthen, 
1985, Adamson and Frick, 2003, Nadel et ah, 2005, Striano et ah, 2005, Soussignan et ah, 
2006, Prepin and Revel, 2007, Auvray et ah, 2009). Certain facial expressions such as joy or 
fear can have direct impact on the partner and induce immediate reaction such as respectively 
smile or fear, certain breaks or modulation within the speech appeal immediate answer. To 
enable partners in an interaction to imitate, to synchronise with each other or even to slow 
down or speed up together, social agents should be endowed with the capability to produce 
such reactive behaviors. 


This dynamical aspect of the interaction is much closer to the lower-level of the agent system 
than to the higher-level of the communicative intentions described by FML: to implement 
this dynamical part of the social agent, the ECA needs reactivity (realtime perception) and 
sensitivity (realtime adapted actions). An architecture enabling dynamical coupling (Prepin 
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and Revel, 2007) has been adapted to GRETA independently from SAIBA framework. This 
architecture implements two capacities giving to the agent dynamical properties and 
reactivity: the self-generation of dynamics (driven by an oscillator) and the sensitivity to the 
partner’s behavior (detected by movement analysis). When two agents interact with each 
other, coupling behaviors such as synchronisation and tum-taking emerge. 


We are presently working on integrating this architecture to the Behavior Planner. The 
Reactive Behavior module will have certain autonomy from the rest of the architecture. It will 
short-cut the Intent Planner, getting input signals directly, i.e. the BML stream coming from 
the analysis of human’s behavior (see Figure 1), as well as the currently planned actions, i.e. 
the BML outputted by the Behavior Planner. 


With these two sources of information, the Reactive Behavior module will propose to the 
Action Selection module two different types of data. On one hand, it can propose adaptation 
of the current behavior by comparing its own actions to the actions of the speaker at a very 
low level. Tempo and rhythm of the partner’s signal production are computed as well as the 
current synchrony between partners. Then a corresponding adaptation of the current behavior 
will be proposed; for example, it can propose to slow down or speed up behaviors. This type 
of proposition may enable synchronisation or similarity of tempo with the user. On the other 
hand, the second type of data proposed by the Reactive Behavior is: By extracting from the 
user’s behavior salient events such as facial expressions, speech prosody or breaks, it will 
propose actions such as performing a backchannel, imitating the user, smiling etc. 
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Finally the Reactive Behavior will be able to propose real-time reactions or adaptations to the 
user’s behavior thanks to its partial autonomy. It will act more as an adaptator of the ongoing 
interaction than as a planner. It is a complementary part of the Intent Planner, much more 
reactive and also working at a much lower level. The EC A must be able to select or to merge 
the infonnation coming from both this Reactive Behavior and the Intent Planner, using an 
Action Selection module. 


Action Selection 

Tyrrell (Tyrrell, 1992) defines the task of an action selection mechanism for an agent as 
“determining, from a set of available conflicting actions, the most appropriate ones”. The 
goal of the Action Selection consists in adapting the actions according to the user’s interest 
level (as it is perceived by the agent) and selecting which action is displayed among the 
possible (conflicting) ones. 


Inspired by the free flow hierarchy approach (de Sevin and Thahnann, 2005), no choice is 
made before the Action Selection module is called. It receives all propositions for action (see 
Figure 1) from the Intention Planner module such as response backchannels specified with 
FML and adaptation to behavior tempo (written with BML) from the Reactive Behavior 
module. Thus, it has to choose between a more cognitive-driven and a more reactive-driven 
behavior. To enable the Action Selection module to make a choice, actions are associated to 
priorities generated in the production modules. The first step is to compute the probability of 
the proposed actions and normalize their priorities according to the user’s interest level. The 
latter is considered as a good indicator of the successfulness of the interaction (Peters et ah, 
2005). 
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Based on this probability unifonnity, the Action Selection module is now able to compare 
action priorities for actions that are conflicting at the signal level. For example, the EC A 
cannot generate a head-shake to mimic the user and a head-nod determined by the 
communicative function “agree” (Poggi, 2005). Only one of these two actions can be 
displayed at the same time and a selection is necessary. The selection is event-based; the 
algorithm is real-time. Finally, the action selection module chooses the most appropriate 
actions based on the priority values according to the user’s interest level. The selected action 
is sent to the FML-to-BML module to be displayed by the ECA. 


Behavior Realizer 


The Behavior Realizer module generates the animation of our agent following the MPEG-4 
format (Ostermann, 2002). The input of the module is specified by the BML language. It 
contains the text to be spoken and/or a set of nonverbal signals to be displayed. Facial 
expressions, gaze, gestures, torso movements are described symbolically in repository files. 
Each BML tag is instantiated as a set of key-frames that are then smoothly interpolated. The 
Behavior Realizer synchronizes the behaviors across modalities. It also solves eventual 
conflicts between the signals that use the same modality. The speech is generated by an 
external TTS 57 and the lip movements are added to the animation. 
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When the Behavior Realizer receives no input, the agent does not remain still. It generates 
some idle movements. Periodically, a piece of animation is computed and is sent to the FAP- 
BAP Player, which avoids unnatural “freezing” of the agent. 


Synchronization 


The synchronization of all modules in the distributed environment is ensured by the Central 
Clock which broadcasts regularly timestamps through the whiteboard. All other components 
are registered in the whiteboard to receive timestamps. 


Interactive applications 


Our agent is designed to be able to manage a natural interaction with users and virtual agents. 
Our first step along this line has been to develop some applications that allow users to interact 
with GRETA. We want to see if our agent can sustain an interaction with a user and how 
users judged the experience. First, we have made the agent perceive the external world: 
Analysis components detecting the user's voice characteristics (like pauses and pitch 
variations) and nonverbal behaviors (such as head movements and facial signals like smile) 
can be connected to our system (Baklouti et ah, 2008; Morency et ah, 2005). This 
information, sent by these components through the whiteboard, is then used to plan the 
agent's response. 


An interactive listening agent 
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Our EGA system has been used to build an interactive listening agent (Bevacqua et al, 2008). 
In the SEMAINE project 58 , we are developing a Sensitive Artificial Listener, SAL, (Douglas- 
Cowie et ah, 2008). Our aim is to endow the agent with the capability to sustain emotionally 
coloured interaction, in particular by showing appropriate backchannels. 



a b 


Figure 3: A real-time listening agent: interaction between a user and the agent in the 

role of: a) Poppy and b) Spike. 


In this project we are considering 4 SAL agents, each defined with a specific baseline 
(Bevacqua et al., 2008). Depending on its baseline and its associated mental state, a SAL 
agent shows very different backchannels. Figure 3 shows a single frame of an interaction 
between a user and 2 SAL agents. The user was asked to tell a story to the agent that shows 
its participation through the production of backchannel signals. 


In Figure 3a, the agent is Poppy an outgoing agent. The agent is showing positive 
backchannels using smile and head-nod. On the other hand, Figure 3b illustrates Spike 
interacting with a user. Spike is very argumentative and uses mainly a frown to signal 
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negative backchannels such as disagreement. In this application, we have interfaced our 
system with Watson (Morency et ah, 2005), a real-time head tracking system. 


Figure4: Architecture of SAL system 


Interactive listening virtual and robotic agents. 


Within the eNTERFACE summer workshop (Moubayed et ah, 2008) which was devoted to 
research on multimodal interfaces, the Listener Intent Planner component has been connected 
with Pure Data (Pure Data), a graphical programming environment for real-time audio 
processing, and with a face detection module that can detect a user's smiles, head-nods and 
head-shakes (Baklouti et ah, 2008). The backchannel signals generated by this system are 
displayed both by our 3D agent and by a SONY Aibo robot (see Figure 4). Since Aibo is a 
dog-like robot, an ad hoc backchannel lexicon had to be elaborated manually. For example, 
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when the backchannel signal requested is a smile the dog wags its tail and bright lights on its 
head and on its back are turned on. The control of the behaviors of the agent and of the robot 
is done via the BML language. The Behavior Planner outputs FAPs for the agent and Aibo 
commands for the robot. 



Figure 4: A user interacts with Greta and AIBO. 


Applications in CALLAS 


Another application of our agent that is being developed within the CALLAS project 59 is 
called The Interactive Storyteller. In this scenario a computer application presents a story 
content and displays a sequence of still images (with high emotional impact) related to the 
story. A web site at the BBC is dedicated to public presentation of news items. Photos 
accompanied by explanatory captions provide emotional content, such as images of the 
earthquake in Schezuan. 
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In the Interactive Storyteller application takes the form of a guided conversation between the 
virtual agent and the user. The ECA is a storyteller. Its role is to interact with the user. It first 
asks her to comment on each displayed image; it then provides some explanation. To enhance 
the emotional experience of the user, the agent shows affective empathy toward her, showing 
its participation in the story. The user expresses her opinion about the images. In the 
background its speech and gestures are analysed by the system to detect her emotional states. 
For this purpose the MKS Keyword Spotting 60 and Gesture Expressivity Recognition 61 
software provided by CALLAS partners are used. Next, the agent explains what the image is 
about. It uses various nonverbal signals like emotional facial expressions and gestures to 
emphasize the message and guide the emotional reactions of the user. Detected information 
about user’s affective state is used by the system to influence the agent’s affective behavior. 
When the ECA begins showing the next scene it expresses the same affective state as 
measured from the user in the previous segment of the story and then, throughout the duration 
of the scene, it gradually changes its expressed emotional state to the target affective state. 


Conclusion 


We have presented an interactive system of a virtual conversational Companion. It models an 
agent while being a speaker and a listener in an interaction. In both roles, the agent’s 
behaviors are derived from its own communicative intentions and emotional state. The agent 
is endowed with expressive capabilities; it can communicate its mental and emotional using 
various modalities (facial expression, gaze, gesture, and head). Behaviors can be of different 
varieties to simulate the agent’s general behavior tendency. Being able to display appropriate 
backchannels allows the agent to show its engagement in the interaction (or its 


238 



disengagement). Reactive responses are triggered by external and internal events, and these 
actions arise from the dynamism between both interactants. They allow synchrony and even 
mimicry to emerge and to be a sign of strong engagement between the conversational 
Companions. Thus expressivity, responsiveness and reactivity are importance features to 
embed in a virtual Companion. 
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A World-Hybrid Approach to a Conversational Companion for 
Reminiscing about Images - Roberta Catizone, Simon Worgan, Yorick Wilks, Alexiei 
Dingli and Weiwei Cheng 


Introduction 


COMPANIONS is an EU project that aims to change the way we think about the 
relationships of people to computers and the Internet by developing a virtual conversational 
'Companion’. This will be an agent or 'presence' that stays with the user for long periods of 
time, developing a relationship and 'knowing' its owners preferences and wishes. The 
Companion communicates with the user primarily through speech, but also using other 
technologies such as touch screens and sensors. 


This paper describes the functionality and system modules of the Senior Companion, one of 
two initial prototypes built in the first eighteen months of the project. The Senior Companion 
provides a multimodal interface for eliciting and retrieving personal information from the 
elderly user through a conversation about their photographs. The Companion will, through 
conversation, elicit their life memories, often prompted by discussion of their photographs; 
the aim is that the Companion should come to know a great deal about its user, their tastes, 
likes, dislikes, emotional reactions etc, through long periods of conversation. It is a further 
assumption that most life information will be stored on the internet (as in the Memories for 
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Life project: http://www.memoriesforlife.org/) and we are linking the SC directly to photo 
inventories in Facebook (see below). The overall aim of the SC as producing a coherent life 
narrative for its user from these materials, although its short term goals are to assist, amuse, 
entertain and gain the trust of the user. The Senior Companion uses a hybrid approach to 
dialogue management as well as intelligent adaptation of the user’s emotional state which 
plays an important part in gaining the user’s trust. 


The technical content of the project is use a number of types of machine learning (ML) to 
achieve these ends in original ways, initially using a methodology used by the Sheffield 
University Companions group in earlier research: first, by means of an Information 
Extraction (IE) approach to deriving content from user input utterances; secondly, using a 
training method for attaching Dialogue Acts to these utterances (Webb, et ah, 2008) and 
lastly, using a specific type of dialogue manager (DM) that uses Dialogue Action Forms 
(DAF) to determine the context of any utterance, and a stack of these DAFs as the virtual 
machine that runs the dialogue and models by means of shared user and Companion initiative 
(REF). The SC is not a robot and could be embodied in a screen, a handbag or a mobile 
phone %% while retaining the same “personality”: it is more a very high level internet agent, 
dedicated to a single user over a long term. In what follows we shall: 

1. Describe the current SC prototype’s functionality; 

2. Sketch its architecture and modules, focusing on the Natural Language Understanding 
module and the Dialogue Manager. 

3. Set out our short term plans to enhance Dialogue Management perfonnance with 
direct Internet access and initial ML experiments. 
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The Senior Companion System 


The Senior Companion (SC) prototype (Wilks 2007, 2008; Wilks et al., 2008) is designed for 
quick advance over the first project year so as to be basis for a second round of prototypes 
embodying more advanced ML; it contains the functionality described in what follows, 
which includes news and photographs. 


The central function of the SC is engaging the user in discussion about their photographs : 
where and when they where they were taken, details about the people in them and their 
relationship to the user and each other. The SC keeps a record of the user’s input and is able 
to pick up discussion with the user where the system left off in later user sessions. In addition 
to reminiscing, the SC also allows the user to do basic photo management including photo 
selection by pointing and grouping of photos through dialogue. 


Once a photo is loaded, it is processed with face recognition software to identify any faces in 
it. This recognition software, OpenCV, provides positional information by identifying the 
face coordinates and this information is exploited in the Dialogue Manager by making 
reference to the position of those in the photograph (person on the left, right, center, etc.) as 
well as recognizing when there are groups of people. The system discusses properties of the 
photo as well as properties and relationships of the people in the photos. 
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The news reading feature adds an interesting accompaniment to the photo domain and 
demonstrates the ability of the system to handle more than one kind of application at a time, 
and has, of course, an unconstrained vocabulary . The news, taken via RSS feeds from the 
BBC news website, includes news from three popular categories : politics, sports and 
business, and the user can choose between them, stop and start the feed by speaking etc. 

This basic system provides the components for rapid future development of the SC, as well as 
its main use as a device to generate more conversation data for machine learning research in 
the next project phase. The system architecture and modules are described briefly below. 

The Senior Companion : 


• Includes a visually appealing multi-modal interface with a character avatar to mediate 
the system’s functionality to the user. 

• Interacts with the user using multiple modalities - speech and touch. 

• Includes face detection software for identifying the position of faces in the photos. 

• Accepts pre-annotated (XML) photo inventories as a means for creating richer 
dialogues more quickly. 

• Has the ability to engage in conversation with the user about topics within the photo 
domain: when and where the photo was taken, discussion of the people in the photo 
including their relationships to the user. 

• Contains a feature for reading news from three categories: politics, business and 
sports. 

• Contains a feature for telling jokes taken from an internet-based joke website. 
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• The system retains all user input for reference in repeat user sessions, in addition to 
the knowledge base that has been updated by the Dialogue Manager on the basis of 
what was said. 

• Contains a fully integrated Knowledge base for maintaining user information which 
contains: 

o Ontological infonnation which is exploited by the Dialogue Manager and 
provides domain-specific relations between fundamental concepts. 

o A mechanism for storing information in a triple store (Subject-Predicate- 
Object)-the RDF Semantic Web format— for handling unexpected user input 
that falls outside of the photo domain, e.g. arbitrary locations in which photos 
might have been taken. 

• Contains a reasoning module for reasoning over the Knowledge Base and world 
knowledge obtained in RDF from the Internet. 

• Contains basic photo management capability allowing the user in conversation to 
select photos as well as display a set of photos with a particular feature. 



Figure 1 : Senior Companion Interface 
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System Architecture 



Figure 2 : Senior Companion system architecture 


The above figure is the Senior Companions architecture. It contains three abstract level 
components - Connectors, Input Handlers and Application Services - plus the Dialogue 
Manager. 


Connectors form a communication bridge between the core system and external applications 
The external application refers to any modules or systems which provide a specific set of 
functionalities that might be changed in the future. There is one connector for each external 
application. It hides the underlying complex communication protocol details and provides a 
general interface for the main system to use. This abstraction decouples the connection of 
external and internal modules, makes changing and adding new external modules easier. At 
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this moment, there are two connectors in the system - Napier Interface Connector and 
CrazyTalk Avatar Connector. Both of them are using network sockets to send/receive 
messages. 


Input Handlers are a set of modules for processing messages according to message types. 
Each handler deals with a category of messages where categories are coarse-grained and 
could include one or more message types. The handlers separate the code handling inputs into 
different places and make the code easier to locate and change. Three handlers have been 
implemented in Senior Companions system - Setup Handler, Dragon Events Handler and 
General Handler. Setup Handler is responsible for loading the photo annotations if any, 
performing face detection if no annotation file is associated and checking the Knowledge 
Base if the photo being processed has been discussed in earlier sessions. Dragon Event 
Handler deals with dragon speech recognition commands sent from interface while the 
General Handler processes user utterances and photo change events of the interface. 


Application Services are a group of internal modules which provide interfaces for the 
Dialogue Action Forms (DAF) to use. It has an easy-to-use high level interface for general 
DAF designers to code associated tests and actions as well as a low level interface for 
advanced DAF. It provides the communication link between DAFs and the internal system 
and enables DAFs to access system functionalities. Following is a brief summary of modules 
grouped into Application Services. 
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News Feeders are a set of RSS Feeders for fetching news from the internet. Three different 
news feeders have been implemented for fetching news from BBC website Sports, Politics 
and Business channels. There is also a Jokes Feeder to fetch Jokes from internet in a similar 
way. 


DAF Repository is a list of DAFs loaded from files generated by the DAF Editor. A fresh 
copy of a DAF can be obtained by passing the DAF name to this module. 


NLG is responsible for randomly selecting a system utterance from a template. An optional 
variable can be passed when calling methods on this module. The variable will be used to 
replace special symbols in the text template if applicable. For example, a template utterance 
“How do you know $?” will be returned as “How do you know John?” if passing variable 
“John” when calling generation method of this module. 


Session Knowledge is the place where global information for a particular running session is 
stored. For example, the name of the user who is running the session, the list of photos being 
discussed in this session, the list of user utterance and etc. 


Knowledge Base is the data store of persistent knowledge. It is implemented as an RDF 
triplestore using a Jena implementation. The triplestore API is a layer built upon a traditional 
relational database. The application can save/retrieve information as RDF triples rather than 
table records. The structure of knowledge represented in RDF triples is discussed later. 
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Reasoner is used to perform inference on existing knowledge in the Knowledge Base. 


Output Manager deals with sending message to external applications. It has been 
implemented in a publisher/subscriber fashion. There are three different channels in the 
system - the text channel, the interface command channel and the avatar command channel. 
Those channels could be subscribed by any connectors and handled respectively. 


The Hybrid-World approach 


The original work behind the SC was based on a closed world where the user converses with 
the system. The system instigates further conversation and in so doing elicits the discovery of 
tacit knowledge from the user. Whilst conducting the initial tests, the limitations of this 
approach immediately became evident. As soon as the user spoke with the SC, the 
conversation quickly went through unexpected paths which required more knowledge than 
was stored within the knowledge-base. For example, if an elderly person is speaking with the 
SC about an old photograph taken during WWII, the person would easily recall events of the 
period. Initially, the only approach we had was to make use of a chatbot since the hot could 
easily keep the conversation generic without understanding the specific events. However, 
conversing with a chatbot is not an inspiring experience so we modified our approach to 
make use of information stored on the web. The problem with that is that the web is an open 
world and it is quite hard for an agent to locate information, assimilate it in its knowledge 
base and use it. 
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Our Hybrid-World approach tackles these issues. Initially, it makes use of a closed-world 
where all the information is stored in the knowledge-base. Every utterance is passed through 
the Natural Language Understanding (NLU) module for processing. This module uses a set 
of well-established Natural Language processing tools such as the GATE (Cunningham, et 
al., 1997) system. The basic processes carried out by GATE are: tokenizing, sentence 
splitting, POS tagging, parsing and Named Entity Recognition. These components have been 
further enhanced for the SC system by adding new and improved gazetteers. These include 
amongst others new locations and family relationships. The NE recognizer is a key part of 
the NLU module and recognizes the significant entities required to process dialogue in the 
photo domain: PERSON NAMES, LOCATION NAMES, FAMILY RELATIONS and DATES. 
Although GATE recognises basic entities, more complex entities are not handled. Because of 
this, apart from the gazetteers mentioned earlier and the hundreds of extraction rules already 
present in GATE, about 20 new extraction rules using the JAPE rule language were also 
developed for the SC module. These included rules which identify complex dates, family 
relationships, negations and other information related to the SC domain. The following is an 
example of a simple rule used to identify relationship in utterances such as “Mary is my 
sister 

Macro: REL ATIONSHIPIDENTIFIER 

({Token.category=="PRP$"} | {Token.category=="PRP"} | {Lookup.majorType=="person_firs 
t"}):person2 

({Token.string=="is"}) 
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({T oken.string=="my " } ) :person 1 


({ Lookup.minorTypc==" Relationship"}): relationship 

Using this rule with the example mentioned earlier, the rule interprets person 1 as referring to 
the speaker thus if the name of the speaker is John (which was known from previous 
conversations) it is utilised. Person 2 is the name of the person. This name is recognised by 
using the gazetteers we have in the system (which contain about 40,000 first names). Apart 
from this, it can also refer to a preposition which can be easily disambiguated using the 
anaphora resolver found in GATE. The relationship is once again identified using the almost 
800 unique relationships added to the gazetteer. 


With this information, the NLU module identifies Information Extraction patterns in the 
dialogue that represent significant content with respect to a user's life and photos. The NLU 
module also identifies a Dialogue Act Tag for each user utterance based on the DAMSL set of 
DA tags and prior work done jointly with the University of Albany (Webb et al., 2008). The 
information obtained (such as Mary sister-of John) is passed to the Dialogue Manager (DM) 
and then stored in the knowledge base (KB). The DM filters what to include and exclude from 
the KB. If for example, the NLU module discovered that Mary is the sister of John, the NLU 
knows that sister is a relationship between two people and as such, it is a very important 
piece of information. However, the NLU also discovers a lot of syntactical information such 
as the fact the both Mary and John are nouns. Even though this information is important, it is 
too low level to be of any use by the SC with respect to the user, i.e. the user is not interested 
in the parts-of-speech of a word. Thus, this information is discarded by the DM and not 
stored in the KB. 
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Once the infonnation is filtered by the DM, the KB stores the information. In essence, the KB 
is a long term store of information which makes it possible for the SC to retrieve infonnation 
stored between different sessions. The infonnation can be accessed anytime it is needed by 
simply invoking the relevant calls. The structure of the data in the database is an RDF triple. 
This is why it is more commonly refer to as a triple store. In mathematical terms, a triple 
store is nothing more than a large database of interconnected graphs. Each triple is made up 
of a subject, a predicate and an object. So if we had to take the previous example, Mary 
sister-of John; Mary would be the subject, sister-of would be the predicate and John would be 
the object. If we had to imagine this graphically, Mary and John would be two distinct points 
in a 3D space and the sister-of relationship would be the line (or relationship) that joins these 
two points in space. There are various advantage to using this structure; first, the relationship 
between different objects is explicitly defined using the predicates in the triples. The second 
advantage is that it is very easy to perform inferences on such data. So if in our 
knowledgebase, we add a new triple which states that Tom is the son of Mary, we can easily 
infer (by using the previous facts) that John is the uncle of Tom. 


Uncle Inference Rule: 

(?a sisterOf ?b), 

(?x sonOf ?a), 

(?b gender male) -> (?b uncleOf ?x) 
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Triples: 


(Mary sisterOf John) 

(Tom sonOf Mary) 

Triples produced automatically by ANNIE (the semantic tagger): 
(John gender male) 


Inference: 

(Mary sisterOf John) 

(Tom sonOf Mary) 

(John gender male) 

-> 

(John uncleOf Tom) 


This kind of inference is already used by the SC and in fact, we do have about 50 inference 
rules aimed at producing new data on the relationships domain. This combination of triple 
store, inference engine and inference rules makes a powerful system which mimics human 
reasoning and thus gives the SC a sense of intelligence. For our prototype we are using the 
JENA Semantic Web Framework for the inference engine together with a MySQL database 
as the knowledgebase. 
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Even though this approach is very powerful, it is still not enough to cover all the possible 
topics which can crop up during a conversation. So in such circumstances, rather than 
switching over to a chat-bot, the DM switches to an open-world model and instructs the NLU 
to seek further information online. 


The NLU makes use of different approaches to achieve this. When the DM requests further 
information on a particular topic, the NLU first checks with the KB whether the topic is about 
something known. At this stage, we have to keep in mind that any topic requested by the DM 
should be already in the KB since it was preprocessed by the NLU when it was mentioned in 
the utterance. As an example, if the user informs the system that the photograph was taken in 
“Paris”, the utterance is first processed by the NLU and then sent to the DM. Once the DM 
requests further information about “Paris”, the NLU goes through the KB and retrieves any 
triples related to “Paris”. Typically, ANNIE (A Nearly New Information Extraction engine), 
our semantic tagger, would have already identified “Paris” as a location and this information 
would be stored in the KB. If it is not found, the semantic tagger analysis the topic and 
provides the NLU with the missing infonnation. 


Once the type of the information is identified, the NLU would use the various strategies 
predefined inside it. In the case of locations, one of these strategies would be to seek for 
information in Wiki-Travel 62 or Virtual Tourists 63 . The system already knows how to query 
these sites and interpret their output by using predefined wrappers. A wrapper is essentially a 
file which describes where a particular piece of information is located. This is then used to 
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extract that information from the webpage. So a query is sent online to these sites and the 
information retrieved is stored in the triple-store. This information is then used by the DM to 
generate a reply. In the previous example, the system manages to extract the best sightseeing 
spots in Paris. The NLU would then store in the KB triples such as [Paris, sight-seeing, Eiffel 
Tower] and the DM with the help of the NLG would ask the user “I’ve heard that the X is a 
very famous spot. Have you ever seen it while you were there?” Obviously in this case, X 
will be replaced by the “Eiffel Tower”. 


On the other hand, if the topic requested by the DM is unknown or the semantic tagger is not 
capable of understanding the semantic category, the system makes use a nonnal search 
engine. A query is sent to these search engines and the top pages are retrieved. These pages 
are then processed using ANNIE and the different attributes are analysed. The standard 
attributes returned by ANNIEinclude information about Dialogue Acts, Polarity (i.e. whether 
a sentence has positive, negative or neutral connotations), Named Entities, Semantic 
Categories (such as dates and currency), etc. The system then filters the information collected 
by using generic patterns and generates a reply from the resultant infonnation. So if the user 
is talking about cats, the system searches for cats online. It processes the pages and its current 
strategy is to identify all the statements by using Dialogue Acts. So in our example, the 
system would retrieve the following statements: 


• Cats may be the most popular pet in the world 

• Cats recover quickly from falls 

• Some people don’t like Persian Cats 
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These statements are then checked for polarity and only the most prevailing statements are 
kept (i.e. if the statements are prevailingly negative then the system will give a negative 
answer, so on and so forth). In this example, the first two statements are prevailingly positive 
because of words such as “popular” and “recover” so the answer returned will be a positive 
one. The NLU would then select one of these two statements at random, send it to the DM 
and using the NLG, it would reply “You know that I’ve heard that X” where X is replaced 
with “cats may be the most popular pet in the world”. 


In synthesis, this hybrid world approach allows us to focus on the closed world that exists 
between the user and the system but when necessary, the system is allowed to venture 
cautiously in the open world thus enriching the user experience. 


Ensuring Persistence of Personality through Intelligent Adaptation 


Traditionally, spoken dialog systems have been able to learn and reason over uncertainty 
through a partially observable Markov decision process (POMDP) that uses reinforcement 
learning (RL) to modify its strategy according to some reward function (Young, 2002). In 
restricted domains with a clearly defined goal (e.g. placing a pizza order) RL proceeds by 
offering a large reward when the task is successfully completed. In the open-ended scenarios 
proposed by the COMPANIONS project how do we capture this reward function? We 
propose that the Companion should be tasked with maintaining the users positive emotional 
state, ensuring continued user satisfaction. Given the goal of maintaining this state emotion 
becomes a central motivating factor requiring an internal representation of both itself and its 
user. 
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We considered two proposed representations of emotion, the discrete theory of basic emotion 
(Ekman, 1999) and the continuous theory, which maps a range of emotions onto a two 
dimensional space (Wundt, 1916, Cowie, Douglas-Cowie, Savvidou, & McMahon, 2000). 
We have settled upon the continuous theory, as instead of arbitarily defining a number of 
descrete emotional categories we can attempt a data driven approach. By creating a self 
organising map (SOM) (Kohonen, 1982), capturing the variability of emotional speech, we 
can train the system on a wide range of emotional utterances forming distinct attractors 
within the defined emotional space. Having trained the system these attractors will then be 
tied to a subset of dialouge acts, reflecting the agents current emotional state. RL can then 
proceed by allowing the Companion to learn how its own movement in this space affects the 
user. However, this is only possible if the Companion can place a representation of the state 
of the user into this space, accordingly we need to assertain the users current emotional 
disposition. 


We propose to achieve this by extracting emotional information from a number of 
multimodal inputs. By combining the semantic content and acoustic information of the 
current utterance the uncertainty of the current emotional state can be reduced (Lee, 
Narayanan, & Pieraccini, 2002, Lauria, 2007). Having established an understanding of the 
users emotional state the Companion will then have a number of ‘tools’ with which to 
manipulate this scenario. 
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The Companion will converse with the user through a number of conversational levels, the 
tone and paralinguistic nature of the utterance can be adjusted to convey emotional content, 
the nature of the dialogue act can be similarly adjusted and the overall conversational strategy 
can be optimised. When exploiting these features it is important to maintain a consistent 
personality that accurately conveys the capabilities and limitations of the Companion; any 
gap between user expectation and reality would create an obvious source of user 
dissatisfaction. As shown by previous WOZ studies (Moore & Morris, 1992) simple 
differences to tone of voice or appearance can cause dramatic differences in user behavior. 
Accordingly, the Companion will present a unified personality (encompassing appearance, 
voice, personality and behavior) which accurately manages the user’s expectation. 


Conclusion and Future Work 


The SC prototype utilizes a hybrid-world approach which enriches the interaction between 
the user and the system. The features mentioned above have been implemented and tested. 
This approach proved to be superior to the closed world approach because it does not limit 
the system to the information inside its own databases derived directly from the user in 
conversation; the general aim here is to break out of the classic AI paradigm of reasoning 
with strong systems over limited closed worlds. We are aiming at a system with weak 
representation and reasoning (i.e. RDF). On the other hand, it is a manageable approach and 
does not suffer from the problems which an open world model would have. Our approach is a 
lazy approach whereby a closed-world model is used if necessary; our agents are allowed to 
venture beyond our systems onto the open world and harvest infonnation which might be 
useful for the progress of the conversation. 
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What we have so far is simply an initial platform on which to build something more 
interesting and complex during the latter half of the project with our partners. During the first 
two years a number of prototypes have been developed and evaluated and the SC is just one 
of those. The Consortium will now build an integrated demonstrator with at least the SC 
functionality we have described; hopefully adding in during the third year full face 
recognition and using machine learning to give a much more flexible barrier between the 
closed (comfortable) and open (uncomfortable) worlds. Moreover, we are working on a 
strong emotional and cognitive models (based around Ekman, 1999, Wundt, 1916, Cowie, 
Douglas-Cowie, Savvidou, McMahon, Sawey and Schroder, 2000) so that the state of the 
Companion can be expressed both in language and other modalities. When dealing with 
understanding problems, this Companion will be able to interface with both closed or open 
worlds thus exploiting the best of both worlds. 
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Companionship is an emotional business - Roddy Cowie 


Abstract 


This is written from the perspective of someone who was trained as a psychologist, and has 
been working for a decade on emotion-oriented/affective computing. That background 
highlights two kinds of issue: how emotion enters into the Companion scenario, and how 
computing can relate to emotion. In both areas, there is a difference between the intuitions of 
people who are not deeply involved, and the realities as they appear to people working in the 
area. The goal of this paper is to consider how the realities of emotion and emotion-oriented 
technology impact on the prospects for artificial Companions. The concern behind it is that 


otherwise, we may misjudge both the prospects and the risks. In particular, ability to address 
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the emotional side of Companionship may play a key part in acceptance; and the necessary 
resources, conceptual as well as technical, cannot be taken for granted. We should be 
concerned about inserting Companions into emotionally sensitive roles without engineering 
them to take that into account. 


The word ‘ Emotion ’ has multiple senses in everyday use. In one, it refers to brief, intense 
episodes (I have called these emergent emotions). In another, it refers to something that 
colours most of life, except for brief times when we are unemotional (I have called that 
pervasive emotion). Emergent emotions may or may not be important for artificial 
Companions; pervasive emotion certainly is. It is presumably related to the biologically- 
rooted ebb and flow of feeling that tends to be called affect, but nobody knows how. 
Engaging with emergent emotions and affect is not dealing with pervasive emotion. 


Pervasive emotion is bound up with the feeling that things and people matter at all (‘caring’); 
that they are valenced (positively or negatively); that we or they are in control of things; and 
that we have grasped or failed to grasp what is going on. It also orchestrates rich, 
unpremeditated exchanges of signals between parties; draws us towards, or repels us from, 
certain courses of action; and shapes what we attend to and remember. Most of these have 
counterparts in the domain of propositional cognition. However, felt sense of significance, 
valence, power, and so on are not the same as our intellectual sense of the same things, and 
the two are not necessarily in agreement. Engaging with pervasive emotion is engaging with 
the fact this felt layer of appraisal and communication exists, and matters. 
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Human-computer interfaces were traditionally oriented towards people who could be 
expected at least temporarily to adopt a particular, more or less unemotional stance to the task 
in hand. Research on artificial Companions cannot rely on that strategy, for several reasons. It 
is an issue how much of life the device impinges on, because it is hard to be constantly 
required to switch into unemotional mode. The populations who need artificial Companions 
are likely to find it hard to adopt and sustain unemotional stances. Communication is at the 
core of both points. Spontaneous communication is emotionally coloured, and producing 
‘careful communication’ that contemporary machines can handle is neither effortless, nor 
possible for everyone. Not least, the Companion’s goals are likely to be bound up with 
emotion - as much to do with making somebody feel happy and confident, as with 
accomplishing practical tasks economically. 


It should be a given that research on artificial Companions is sophisticated about issues like 
these. Ignorance about emotion is not excusable. That is not trivial, because at present there 
are very few sources that the area can turn to for sophisticated information about them. 


It is a very difficult challenge to develop technologies relevant to engaging with emotion, and 
public understanding of the state of the art is quite limited. Among other things, that means 
that people may be worried or enthusiastic about developments that are very unlikely to 
occur, and unaware of problems or opportunities that current technology brings quite close. 


Among the commonest misunderstandings is the idea that engaging with emotion is 
necessarily about devices that mimic human emotional life. The basic foundation is design 
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that takes human emotionality into account in a systematic way. That is an unquestioned part 
of many areas of human interest - entertainment, clothing, the car industry, and so on. It is 
quite intriguing that it should play so little part in so much of computing, and still more so 
that there is active resistance to bringing it into play. That basic level of attention to emotion 
cannot be left out of work on Companions. 


An almost equal and opposite fallacy is that we know there is a fundamental obstacle to the 
acceptance of devices that are too human-like - the memorably named ‘uncanny valley’. On 
scrutiny, though, what is known is just that some very human-like devices are very disturbing 
- in the classic study, a prosthetic ann and a zombie. It is not known whether these are signs 
of a long valley lying in wait for any approach to human-like qualities, or isolated potholes 
that a sensible team will steer round. After all, a zombie would perhaps not be the first choice 
of model for most teams setting out to build a Companion. 


Overall, convincing evidence on the emotional appeal and/or repulsion of various kinds of 
device is in short supply. That is partly because it is very hard to assess the emotional impact 
of a product (be it a Companion or anything else) that engages with people in a complex way 
over long time periods. One of HUMAINE’s work packages has worked extensively on the 
problem, both developing new techniques and reviewing the available options at length. 


The human-like emotional abilities that a Companion might incorporate divide into three 
areas - detecting emotion-related signals that a person gives, synthesising its own emotion- 
related signals, and carrying out emotion-related planning and reasoning. 
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Probably the most developed of these areas is synthesis of visual signals. Starting from 
representations that specify the internal states and intentions to be expressed, artificial agents 
can generate appropriate facial signals, eye movements, hand gestures, and body movements 
in increasingly co-ordinated ways. Paradoxically, this is the ability that it is hardest to 
imagine Companion-type systems using. On the other hand, synthesis of speech signals has 
proved much more difficult than might have been expected. Techniques that allow good basic 
quality do not give the flexibility to synthesise the movements needed to express emotion, 
and vice versa. Synthesis of actions that express emotion - banging a cup down or hugging 
someone - is rarely considered. 


Detection is effectively a step behind. In synthesis, the signs associated with a state can 
unfold in a structured way over time. Recognition is currently rooted in a synchronous 
paradigm - signs in a given interval are mapped onto state in the same interval. That may be 
appropriate for strong emergent emotions; it is a very coarse tool for monitoring pervasive 
emotion. We take it for granted that a person can detect mild emotional colouring in another 
person’s demeanour - irritation, distress, interest of lack of it. We take it for granted that even 
animals have some sensitivity to states like that. There would be considerable implications 
for the way a Companion could function if it was not able to detect those signals, even when 
person involved believed - reasonably - that he/ she was making them very clear. 


The need for emotion-related planning and reasoning follows on very directly. If a 
Companion can detect boredom, irritation, happiness, and so on, it has then to decide what to 
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do about it. Simple prescriptions like ‘match the state’ go some way, but not very far. The 
human default is to match some emotions (such as happiness), but not others (such as grief). 
In a case like anger, it matters crucially what the person is angry with. The response to an 
unwelcome emotional state is often to find something that might alleviate it, and here another 
kind of reasoning is needed, probably involving personalised learning about what might 
soothe a particular person’s irritation or sorrow. Note that it is not obvious whether these 
responses should involve signals of emotion as such. It might well be effective to suggest an 
appropriate activity in a neutral way. 


It is sometimes proposed that that kind of planning must involve a degree of empathy - which 
implies a simulation of emotion within the agent. Two points should be made about that. 
Technically, it is not at all obvious that it is true. Propositional (‘BDI’) frameworks are being 
adapted to provide a degree of emotional reasoning, and people who work with BDI networks 
are not likely to attribute much emotion to them. Experts should resist the suggestion that 
engaging with people’s emotions means creating Companions that have their own. The 
second point, though, is that lay people may well react very differently, and have a 
compelling sense that a Companion is empathising with them when it offers just the right 
suggestion to alleviate a sad or angry mood. 


That leads into a vexed area, which is the ethics of engagement with emotion. HUMAINE 
has undertaken to provide a ‘white paper’ on the subject which tries (as this paper has done) 
to address the real issues that genuinely may become real. Three deserve particular mention. 
The first is deception. What is the ethical status of building a device whose behavior signals 
emotions that it does not - in any straightforward sense - actually feel? The problem is 
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redoubled if the person involved moves from assuming - perhaps tacitly - that the 
Companion does have wann emotions towards him/her, to a sense that he/she has been 
betrayed. The second issue is the ‘lotus eater’ problem: making a Companion too 
emotionally engaging risks eroding the person’s motivation for engaging with human beings, 
who - notoriously - are not always emotionally engaging; and therefore increasing isolation 
instead of reducing it. 


The third issue hinges on the likely technical limits of detection. For the foreseeable future, 
machines’ ability to recognise emotion-related attributes will be modest. Misattributions will 
happen. There is a real problem if these misattributions influence what happens to the person 
- for instance, false attributions of negative mood feed into the medical system; false 
attributions of hostility feed into a care system; and so on. These are special cases of the 
problems that arise with ‘SI IF’ systems - semi-intelligent information filters. They are not 
unique to systems that attempt to detect emotion, but emotion detection raises them very 
acutely because people are so prone to underestimate the complexity of detecting emotion. 


In conclusion, it is crucial to recognise that most of the issues raised above are not created by 
engineers choosing, Frankenstein-like, to create an artefact with pseudo-emotional qualities. 
They arise willy-nilly when artefacts are inserted, into situations that are emotionally 
complex, to perfonn a task to which emotional significance is normally attached. There is a 
moral obligation to attend to them, so that if we intervene in such situations, we do it in ways 
that make things better, not worse. 
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The HUMAINE portal, http://emotion-research.net/ , contains reports which provide detailed 
information on and discussion of issues covered in this paper. 


Artificial Companions in Society; Consulting the users - Alan Newell 


Abstract 


Artificial Companions provide Companionship, and to help people with their daily tasks is an 
intriguing and exciting concept. In addition to the important technical challenges of 
providing such systems, it is also clearly important to consider: 


1. What facilities they should and should not provide. 

2. The characteristics, needs and wants of potential users and 

3. The personal and social consequences of such technologies. 
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These are complex, multi-faceted issues and it is necessary consider what are the most 
effective and beneficial ways of addressing them. 


Introduction 


One approach to the considerations above is the use of interdisciplinary fora. These would 
include a variety of experts and be facilitated in such a way as to encourage wide ranging 
discussions of Companion technology, what it can achieve, and the positive and negative 
aspects of the introduction of such technologies. The danger of using groups of experts, 
particularly when they are dominated by technical people, is that the focus can come to lie on 
designing and building what experts think the target population needs and wants. These 
discussions can also be influenced by what the experts find most exciting and interesting to 
develop, and which they judge will be most effective in magnifying their reputation and/or 
profits. There are many Infonnation Technology systems which exhibit these characteristics: 
Software with a huge range of functionality which the vast majority of users neither want nor 
need, and where this excessive functionality leads to very complex systems, which are 
difficult to leam to use. Technical experts are also not always the best people to predict the 
future requirements for technologies. It is interesting, for example to compare the grossly 
over-optimistic predictions concerning speech recognition systems since the 1960’s with text 
messaging not being deliberately offered as a facility of mobile telephones. 


The Characteristics of Potential Users 


267 



If we are to develop successful Artificial Companions, it is very important to consider the 
characteristics of potential users of artificial Companions as well as the technology. 
Particularly in the case of those potential user groups who are most in need of such 
Companions - older and disabled people - these can be very different from the characteristics 
of designers of new technologies and their traditional user base. Older people have multiple 
minor disabilities, sensory, motor, and cognitive, and may also have major disabilities. The 
vast majority will also have substantially different experiences of, and emotional attitudes to, 
new technologies than younger cohorts. Many of them will not understand, or be familiar 
with the jargon, metaphors, or methods of operation of new technologies, and have major 
lack of confidence in their ability to use them (Newell & Gregor 1997 & 2002). It is often 
suggested that, when the current cohort of middle aged people become old, these challenges 
will no longer exist as these users will already be very familiar with computers. This na'ive 
view assumes that the physical and mental characteristics of these people will not be affected 
by their growing older. It also assumes that human interface methods and metaphors will 
forever remain static, but the history of information technology shows that such an 
assumption is unlikely to be true. In addition, it is important to note that a significant fraction 
of currently middle-aged people - mainly, but not exclusively those with low educational 
attainment - are computer naive. Thus the challenges of providing IT systems which are easy 
for older people to use will not go away. 


If we are to develop successful artificial Companions for older and technologically naive 
users we need to address such issues as: 
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a) What are the physical, sensory, and cognitive characteristics of major sections of the 
potential user base of artificial Companions? 

b) What are the technological experiences of, and attitudes to, current technologies of this 
user base? 

c) What facilities would this group of people like to have in their artificial Companions, and 
what safeguards would they require? What are their needs and wants for artificial 
Companion technologies? 


We need to be aware that cohorts of older people are likely to have entirely different 
responses to such questions than younger and/or technologically experienced people. 


Requirements Gathering and Knowledge Elicitation 


Any requirements gathering exercise about Artificial Companions needs to involve iterative 
discussions with older users, in such a way to facilitate both users and designers to be 
creative. It is thus necessary to consider what the most effectively ways of including the 
experiences and views of potential users, particularly elderly and technologically naive users 
in meaningful and creative discussions about artificial Companions? 
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A number of techniques have been developed to encourage designers to interact with their 
potential user base in an effective and efficient manner, such as User Centred Design, 
Participatory Design, and Ethnography. 


User Centred Design encourages designers to focus on the users within the design process, 
and suggests ways of doing this (Preece, 1994, Shneidennan, 1992, Newman & Lamming, 
1995, and Helander et al. 1997). Participatory Design, (Muller, 2002 and Kyng and 
Mathiassen, 1997) is a design technique which proposes a focus on the users, and 
Ethnography involves users within the design process (Blomberg et al. 2002), with the 
ethnographer acts as an intermediary between the users (usually in their own environment) 
and the designers. 


These methodologies, however, rarely fully address the ethical and pragmatic issues 
involved when the user groups consists of elderly and possibly very frail people - some of 
whom may have serious cognitive dysfunction. These can impose significant changes to 
these design techniques and needs to be carefully considered (Newell & Gregor 1997, 
Newell et al 2008). 

Questionnaires 


Even when questionnaires are carefully constructed, they can elicit misleading and grossly 
incorrect data. Age related cognitive deficits can make self reporting inaccurate, and age 
differences affect the ways in which people respond in self-reports. Older respondents use the 
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“don’t know” response more than younger respondents, and in response to questions that 
have complex syntax or are semantically complex. Park and Schwarz (2000) report that older 
respondents are generally also more “cautious” in their behavior, and need to “have higher 
threshold levels of certainty” before responding to questions. 


Focus Groups 

There can be communication challenges in running focus groups with older people, caused by 
decreasing abilities in sight, hearing and short term memory and other cognitive processes. 
Keeping focus groups of older people focussed is not easy. Researchers have reported on the 
problems encountered when running focus groups with older people (Lines and Hone, 2002; 
Barrett and Kirk, 2000). In addition, Eisma et al. ( 2003a) have found that older people, in 
can be reluctant to complain, or criticise products. They can experience more computer 
anxiety, and be more negative about the amount of effort required to learn to use them. 

These characteristics are often fuelled by the assumption by older people that they have no 
use for information technology systems (Marquie et al. 2002). 

Technological Naivety 


There are difficulties in orientating technologically naive people to the challenges and 
promises of new technology, and people, particularly older ones, can lack technological 
imagination. Discussions about new technologies can be likened to asking people in the 
middle of the 18 th Century about the characteristics of horseless carriages or telephones, or in 
the early 1980s about home computers or mobile telephony. Thus older people need to be 
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presented with accurate instantiations of the technology being discussed, if they are fully to 
understand what is being asked of them. 

Multiple minor disabilities 

A further challenge concerning human interfaces to infonnation technology is that older users 
of computer systems can be affected by multiple disabilities, and such multiple minor (and 
sometimes major) impairments can interact to produce a handicap that is greater than the 
effects of the individual impairments. Commonly used techniques which are focused on 
single impairments may not always provide appropriate solutions for older users. 

Sensitive Issues 

Particularly important for research into Artificial Companions is older people’s reluctance to 
discuss issues which reflect badly on their own self image or which they perceive will effect 
other people’s perceptions of them. For example, Russell (1999) found that many older 
people may not want to talk about sensitive topics such as social isolation, “because such an 
acknowledgement challenged their identity as independent people”. 


User Sensitive Inclusive Design 


A range of techniques have been suggested to overcome some of these issues. Newell (2008) 
suggested an approach called User Sensitive Inclusive Design. Eisma et al (2003b) proposed 
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the concept of mutual inspiration to avoid the researcher /designer divide which often 
threatens the effective communication of requirements to the design team. Hands-on 
sessions, where older people experience new technology, have also proved more successful 
than verbal explanations or demonstrations, and these can often lead to spontaneous 
suggestions for improvements or for new products (Inglis et al, 2002). Where appropriate 
these can include Wizard of Oz Techniques to give the audience a clear idea of how a 
proposed system might actually work. 


A further possibility is the use of theatrical techniques. These have been found to be 
successful in facilitating dialogue between designers of future technologies and older people, 
and releasing the creative intelligence of the potential user population. 

The use of theatre 


Forum Theatre techniques in particular encourage substantial interaction between the 
audience and the actors about the particular issues addressed by the theatrical presentation. A 
script writer conducts detailed research on the subject area and produces a series of short 
plays which address the important issues to be discussed (Newell and Morgan 2006a). They 
are in a narrative rather than documentary form, and contain ‘human interest’, humour, and 
dramatic tension as well as illustrating how the system may work, the errors which could 
occur in its use, and the effects of these errors on the participants. Each scenario lasts 
approximately five minutes, and typically leads to about twenty minutes of discussion. The 
format can be a video presentation followed by discussions, or live theatre, or a combination 
of the two. 
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These techniques have been used within the requirements gathering phase of a project 
developing a video camera based ‘fall ’ monitor and detector for older people in their homes 
(Marquis-Faulkes, et al. 2005), and produced lively and very useful discussion with older 
people on the characteristics of such systems. The primary aim of the UTOPIA (Usable 
Technology for Older People: Inclusive and Appropriate) project, was to develop techniques 
for changing the mind sets of designers concerning the needs of older people. It culminated 
in the production of the UTOPIA Trilogy, a series of videos addressing issues of older 
people’s use of technology (Cannichael et al 2005).Evaluations of these video with students 
and professional designers have shown that they are successful in raising awareness of the 
challenges older people have with new technologies. The film “Relative Confusion ” has been 
developed to illustrate the challenges older people have with the introduction of digital 
television and a further film is being produced to address the wider issues of the exclusion of 
older people from the digital economy. Live theatre and a combination of video and live 
theatre have been presented at a number of international conferences, and elicited very 
interesting and useful discussion with the audiences. (Newell and Morgan 2006b and Morgan 
and Newell 2008). 


Rice et al (2007) describe the advantages for very novel design briefs where an entirely new 
technology is being developed. It is worth noting that script writers and actors are trained as 
professional observers of human behavior and their skill is presenting that behavior in a way 
which engages the viewer/audience. The use of actors removes the ethical problems of 
“protecting the users”. It is possible to envisage a situation where the designers and the users 
can verbally attack one another as part of addressing the usability issues of a particular 
system, a situation which would be difficult traditional usability laboratory setting, and likely 
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to be unethical. Actors can also present a more generic picture of a user and can change their 
personas in response to requests from the designers (e.g. what would happen if you were 
older, if your sight/hearing was impaired, if you were under pressure?). 


Finally, theatre encourages a creative approach to design, involving users as well as 
designers, rather than the traditional view of focus groups and usability testing being solely a 
method for eliciting users’ views and opinions, and detennining their abilities to use specific 
interfaces and systems. The use of professional actors and live theatre has been shown to be 
effective in facilitating discussions with users, or in those cases where it is too difficult or 
inappropriate to involve real users in experiments and dialogues (Newell, Cannichael et al. 
2006). Although not an inexpensive option, the value which can be brought to the 
interaction by theatre professionals can be very significant. 


Conclusions 


Any research into the present and future of Artificial Companions needs to include a 
consideration of the characteristics, needs and wants of the user group, and on the most 
effective ways of eliciting such information from appropriate cohorts of users. A number of 
techniques appropriate for this activity have been developed including the use of narrative 
video and live theatre. It is recommended that people seeking to research into and develop 
Artificial Companions for older and disabled people examine these techniques in detail and 
use those which are appropriate to their particular requirements. 
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Requirements for Artificial Companions: It’s harder than you think 

Aaron Sloman 

Producing a system that meets plausible requirements for Artificial Companions (AC’s), 
without arbitrary restrictions, will involve solving a great many problems that are currently 
beyond the state of the art in Artificial Intelligence (AI); including problems that would arise 
in the design of robotic Companions helping an owner by perfonning practical tasks in the 
physical environment. In other words, even if the AC is not itself a robot and interacts with 
the user only via input devices such as camera, microphone, keyboard, mouse, touch-pad, and 
touch-screen, and output devices such as screen and audio output devices, nevertheless it will, 
in some circumstances, need the visual competences, the ontology, the representational 
resources, the reasoning competences, the planning competences, and the problem-solving 
competences that a helpful domestic robot would need. This is because some of the intended 
beneficiaries of ACs will need to be given advice about what physical actions to perform, 
what physical devices to acquire, and how to use such devices. I shall give examples 
illustrating the need for such competences. 
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Introduction 


The invitation to write this chapter on Artificial Companions (AC’s), said they would be: 


Conversational software-based artificial agents that will get to know their owners 
over a substantial period. These could be developed to advise, comfort and carry 
out a wide range of functions to support diverse personal and social needs, such as 
to be artificial Companions for the elderly, to help their owners learn or to sustain 
the fitness and health of their owners. 


Moreover, although there are many robot projects aiming to produce useful, more or less 
intelligent, autonomous robots, and some of them 64 are very impressive at performing limited 
tasks (e.g. welding car bodies, or moving around on rugged terrain) they are nowhere near 
meeting the requirements of a helpful multi-purpose Companion. That is because there are 
significant differences in requirements for forms of representation and information¬ 
processing architectures between (a) machines that are merely able to perform complex tasks 
in a physical environment and (b) machines that understand what they have done, what what 
they could do, what they should not do, why they should not do it, what the consequences of 
actions will be, what further options could arise if a possible action were performed, how all 
this relates to what another individual could or should do, and can also communicate some of 
this to other individuals. Insects are very impressive as regards (a) but appear not to have 
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competences of type (b) except in very limited ways (e.g. communicating through pheromone 
trails, or bee dances). 


One of the problems in producing robots with competences of type (b) is that the 
requirements for such systems are extremely complex and subtle, and far from obvious, 
whereas many researchers think that the requirements are obvious and well understood, so 
that the only task is to work out how to produce systems that meet the requirements. Similar 
problems will arise for work on DCs. 

The assumption that the requirements are clear is based on a failure to understand the rich and 
varied details of the processes of perceiving, understanding and interacting with all the 
different sorts of objects and situations that can occur in a normal house, in contexts where 
different sorts of human needs have to be met, many of which cannot be predicted in 
advance, so that solutions will require creative problem-solving by the DC at run time. I shall 
attempt to explain some of those problems briefly below, though more details can be found in 
(Sloman, 1982, 1989, 1996, 2001a, 2001b, 2008a, 2009 Sloman, Wyatt, Hawes, Chappell, & 
Kruijff, 2006, Sloman, 2007, 2008c). One way of expressing the point is that we do not yet 
understand what all the problems were that biological evolution solved, in producing designs 
for infants that can develop into human beings in many different physical and cultural 
contexts. As John McCarthy (2008) noted, evolution did not solve the problem of producing a 
robot-like system that starts with no knowledge about the environment but has some 
completely general, immensely powerful, learning system. 
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One way to avoid these very demanding requirements is to restrict severely the intended role 
of the DC, for instance producing only “engaging” ACs (defined below), and only the simpler 
sorts of “helper” ACs. However, it is important (a) that both researchers in this area and 
potential users of such devices, and their carers, come to understand the variety of 
requirements for helpful ACs, (b) that they understand why some of the requirements are 
very hard to meet, and may remain so for many years, and (c) that people who consider 
selecting, buying, using, or funding development of ACs are not misled into expecting what 
cannot be delivered. 

Terminology: different human roles. 

I use the word “user” to refer to individuals for whom ACs are provided. The word “owner” 
was used in the quotation at the beginning of the chapter,, but we need to distinguish the 
individuals to be helped by ACs from those who buy and own them, such as helping 
organisations, relatives of users, etc. A user of a AC will regularly interact directly with the 
AC, and is expected or intended to derive some benefit from doing so. I shall use the word 
“carer” to refer to other individuals who are involved in the decision to acquire the AC, or 
who have, or feel they have, some ongoing responsibility to the user. Typically carers will not 
interact with the AC, or will do so less frequently than the user. We can distinguish “direct 
carers” who interact regularly with the user and, where appropriate with the AC, in order to 
train it, check its logs, arrange repairs, etc., and “indirect carers” who have some 
responsibility for or concern about the user, but either choose not to be or are unable to be 
closely involved, so that they leave it to others to be direct carers, and in some cases 
contribute to the cost of the service. In some cases the roles will overlap: e.g. the same 
individual might be owner of the AC and direct carer, or owner and indirect carer. 
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Another group of individuals with interests and responsibilities are the “developers” who 
work on producing or improving ACs. They may have a variety of different motives 
(including worrying about what their own quality of life will be in old age! ), though most 
seem to believe they are working to improve the human condition, at least for some humans. 


A AC user’s interests are not always the main or sole concern of developers, owners, direct or 
indirect carers or investors funding the development, and this can raise serious ethical issues, 
for instance when a AC is provided not in order to raise the quality of life of the user, but 
merely to reduce the burden on a carer, or in order to make money for owners, direct carers 
and others. In some cases there is no conflict, e.g. where ACs really do enhance the quality of 
life of users and also provide an income for owners, carers, developers and funders, or reduce 
burdens on carers. However such ethical issues are not the main topic of this paper. 


Types of function for digital Companions 

A great deal of work is currently being done on developing artificial agents (robotic and non- 
robotic) that can interact with humans, as toys, as game participants, as tutors, as counsellors, 
as marketing devices, in various forms of entertainment, and for research on various topics, 
including investigations of affective processes (much of it inspired by Picard (1997)). Not all 
the devices being developed are robots and not all are being developed as long tenn digital 
Companions. Many of them will share requirements with ACs. The target functions of ACs 
and other interactive devices can be roughly divided into two broad categories: (a) ‘engaging’ 
functions that focus on the quality of interaction between machine and user and the ability of 
the machine to hold the attention and interest of the user, keeping the user entertained, 
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amused, or merely interested, independently of any benefit that the user may gain from the 
interaction; and (b) ‘enabling’ functions that focus on meeting needs or achieving goals of the 
user, The division is not very sharp, however. Each of those will be further subdivided, in 
order to contrast the functions that will be hardest to achieve from the rest. 

Engaging functions 

It is useful to identify a collection of functions that many developers currently aim to provide, 
which I call “engaging functions”, e.g. keeping the user happy, quiet, entertained, 
uncomplaining, etc. This can often be achieved by providing fairly shallow competences in 
machines, as illustrated by entertaining chatbots, motorised toys, speaking dolls, or in some 
cases merely cuddly toys. The kinds of devices that can be used to provide engaging 
functions include at least the following sub-categories, though this is not meant to be a 
definitive classification: 


• TOYS: Entertaining devices for occasional use (compare dolls, simple computer 
games, recorded music players, and what could be described as “entertaining dummy- 
humans”, i.e. superficially human-like electronic products that have amusing physical 
behaviors, or provide entertaining verbal or graphical interaction, e.g. some chatbots.) 

• INTERACTIVE DEVICES: Products that are intended to be used regularly to provide 
interest, rich and deep enjoyment through active participation by the user, or a feeling 
of Companionship (compare pets, musical instruments, construction kits, more 
intellectually demanding computer games). 

• DUMMY-HUMANS: Products that are intended to be regarded by a user as being like 
another caring, feeling individual with whom a long term relationship can develop - 
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even if that is based on an illusion because the machine is capable only of shallow 
manifestations of humanity: e.g. learned behavior rules, such as nodding, smiling, head 
turning, gazing at faces, making cute noises, or in some cases apparently asking to be 
comforted, etc. 

Unfortunately, there are some theories of emotion that treat emotions as little more than 
dispositions to produce shallow manifestations that can be recognised as indicating sadness, 
surprise, being amused, anger, fear, relief etc. Researchers who believe such theories often 
assume that they are working on giving machines emotions when all they are doing are 
producing machines that at best either deceive or engage. For a discussion of differences 
between deep and shallow theories of emotions see (Wright, Sloman, & Beaudoin, 1996, 
Sloman, 2001a). 


The engaging functions have probably attracted the most research and development effort, 
since they are the easiest to achieve, and, I suspect, most fun for researchers. Some of this 
work is very superficial, namely getting machines or avatars to produce various mildly 
convincing imitations of human behaviors, without trying to understand or replicate the 
information-processing that leads up to and produces, maintains or modulates those 
behaviors, or reactions to them, in humans or other animals. I worry that some, but not all, 
work on engaging functions is pursued cynically for financial gain though deception of 
potential users, owners, carers, or funders, or else pursued naively because the researchers are 
deceived about the significance of such products, because they believe incorrect theories. For 
a first draft attempt at explaining aspects of grief that contradict most theories of emotion, for 
instance because grief can endure for months or years, and can coexist with short tenn states 
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such as joy, amusement, relief, etc., see (Wright et al.., 1996). The following sections mostly 
ignore work on engaging functions, except by contrasting them with enabling functions. 


Enabling functions 

Future ACs will be developed in order to produce more sophisticated functions than the 
engaging functions, for example providing help, assistance, advice, etc. for users, including 
helping them to do things they would wish to do even if they did not have a AC to help, for 
instance, moving around, getting dressed, reading newspapers, having a bath or shower, 
cooking food, learning some new skill, deciding whether to call a doctor, being reminded to 
take pills, dealing with a minor injury or domestic mishap, obtaining information, etc. Some 
enabling functions can be part of therapeutic treatment, e.g. helping someone who has been 
injured or had a stroke do exercises to recover lost abilities, as in (Tapus, Tapus, & Mataric, 
2008). 

ACs providing enabling functions may have various levels and kinds of competence. The 
following is a rather shallow, sub-division into three successively more sophisticated levels 
or enabling functions. A more substantive analysis would require far more detailed 
subdivisions, e.g. based on types of competence required for different functions. 


• HELPERS: 

These are systems that can reliably provide help and advice that meets practical 
everyday needs as well as occasional unexpected problems, for example: detecting that 
the user has fallen and is not moving, then calling an ambulance service; answering 
questions using a database of frequently asked questions, with answers that are likely to 
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suit all or most questioners; detecting a knock at the door and flashing a light to attract 
a deaf user’s attention; providing advice or answering questions by using an algorithm 
for searching the internet for such questions and their associated answers, and many 
more. These systems provide help on the basis of explicitly provided infonnation and 
algorithms, or as a result of being trained in advance on some corpus to detect classes 
of problems and associate them with specific answers from the corpus, or in some 
cases answers derived by simple logical inference or mathematical calculation. Such a 
system will not develop new capabilities unless re-programmed or re-trained. A helper 
that is able to extend its capabilities while on the job will fall into one of the following 
categories. 


• DEVELOPING HELPERS: 

These are Companions that are capable of developing, over time, an increasingly deep 
and broad (perhaps also human-like) understanding of the user’s environment, needs, 
preferences, values, knowledge, and capabilities, so that they extend and improve their 
ability to help and support the user, as opposed to being restricted by the functionality 
given to them by their designers. An example might be noticing that the user is 
becoming more absent-minded and learning to detect situations arising, e.g. when tasks 
are left unfinished, when the user needs a reminder or warning, etc., and learning how 
to provide such advice and help in a manner that is not resented or disliked. Another 
example might be noticing that the user is starting to find certain physical tasks 
increasingly difficult, detecting a pattern in the tasks and the problems, and reporting 
the discovery to a carer. 
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How such learning can occur, what kind of infonnation-processing architecture can 
support it, what physical interfaces are required (e.g. sensors distributed around the 
home) and how much the user has to cooperate if the learning process is to be reliable, 
are all important details that will not be discussed here. 

Learning helpers are needed because it will be impossible for designers to anticipate 
everything and also because the user’s situation and the user’s needs and preferences 
will change over time. More advanced abilities to extend themselves will be required in 
ACs in the next category. 


• CARING DEVELOPING HELPERS: 

A human user and the user’s environment will provide very many opportunities for 
learning, since over time the user may change (e.g. deteriorating physical and/or mental 
health, loss of a particularly close human carer, changed financial circumstances, 
buildings, furniture, or appliances deteriorating, or being replaced by new versions with 
different characteristics, acquisition of new household appliances, and development of 
new interests, hobbies and concerns by the user). 


The AC will have to choose which of these changes in its environment to attend to, and 
make decisions about which observations are relevant to its function. How to control 
such a process is far from obvious, and it is likely to be very difficult. One form of 
control would be to make the helper really want to do what is best for the user. This is 
analogous to biological parents wanting to do what is best for their offspring instead of 
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merely having fixed reactions to situations involving their offspring, selected either by 
evolution or some social pressure group. 


So the most sophisticated ACs may need to be developing helpers that grow to care 
about the user and really want to help when things go wrong or have a significant 
probability of going wrong, and which want to find out how best to provide such 
help. 65 


This sort of caring (real, not simulated) will be required to ensure that the AC is 
motivated to find out what the user wants and needs, and so that it can deal with 
conflicting criteria in ways that serve the user’s interests. 


Such a helper will want to find out what the user wants, prefers, likes and dislikes; and 
will also come to want to act in accordance with those attitudes. It may sometimes 
notice conflicts between what the user wants and what is good for the user and will 
generally favor the long term benefit of the user, 66 making exceptions in the sorts of 
cases where a caring human helper would. 


There is a pattern in these three broad categories that is also relevant to the successively more 
demanding problems “solved” by evolution in producing increasingly cognitively and 
architecturally sophisticated organisms to accommodate increasingly complex, varied and 
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unpredictable environmental demands and challenges (as discussed in Sloman and Chappell, 
2005, Chappell and Sloman, 2007, Sloman, 2007a). 


I have separated the engaging functions from the enabling functions in order to indicate that 
the main focus of this chapter is the enabling functions and how to provide them. I am 
interested only in the problems of producing helpers, especially caring, developing helpers, 
and not at all interested in producing engagers. Unfortunately, producing good helpers, 
especially caring, developing helpers, is much harder than producing shallow engagers, and, 
equally unfortunately, it seems that some researchers ignore the differences, and 
consequently underestimate the difficulties in producing the enabling ACs. 


Later sections will examine some enabling functions of ACs more closely, explaining why, 
for those functions, major advances are needed in both our understanding of requirements and 
in our knowledge of how to design and implement machines with human-like competences, 
as opposed to merely giving the appearance of having such competences through superficial 
behavior in constrained testing conditions. Achieving the ambitious enabling functions will 
require us to solve interdisciplinary research problems involving AI, philosophy, psychology, 
linguistics, biology and possibly neuroscience. That includes acquiring a deeper 
understanding of the problems solved by evolution in producing humans and other animals. 
Recent work by developmental psychologists is relevant, though unfortunately not all of them 
adopt what McCarthy calls “the designer stance” (McCarthy, 2008). 
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Motives for developing, funding, buying or using ACs 


In addition to distinguishing the different kinds of function that can be served by a AC we can 
ask about the reasons or motives behind decisions to acquire one, bearing in mind that 
sometimes the user will not be the decision-maker. Decisions to apply such systems, and to 
choose between different combinations of functions to go into a AC will often not be made 
by the user, or not by the user alone. For example, government or medical agencies, members 
of the family, and managers of homes for such people may all be involved, either acting alone 
or in cooperation with one another. They may or may not take the user’s needs, desires and 
preferences into account, and when they do take them into account they can give the user’s 
needs etc., differing relative weights. 

In particular there are motives that concern only the user’s desires and benefits, and 
motives of other humans who care about or have responsibilities to the user. Example 
motives for providing a AC are: 


• Because the user wants the AC: 

For example, the user may sincerely prefer to be helped by a AC so as not to have to 
impose on other humans and the others involved may respect that preference, even if 
they would prefer to provide the care themselves. 

• Because carers have constraints: 

The others may want the AC to be available to fill gaps and provide needed help and 
care when human carers are unavoidably unavailable, e.g. because they have children 
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they have to look after, or because they need to go to work to earn funds to pay for the 
care, or because they are badly needed elsewhere, etc. 

• Because carers do not care enough: 

The others may wish to use the AC in order to enable them to avoid tasks that they find 
distasteful or because they have other personal preferences/priorities 

• Because carers have theories about needs of users: 

A particular type of AC, like a type of medicine, or a type of home for a user, may be 
chosen by carers because they have theories about what would be “best” for the user. 
Such theories may or may not be accepted by the users in whose alleged interest such 
decisions are taken. Moreover, even if a theory is accepted by everyone involved, it 
may be fashionable but false. 

Obviously these goals and preferences can in some cases be somewhat cynical or selfish: The 
main beneficiaries of a AC in some situations will not be the user but others connected with 
the user, either because of personal relationships or because of contractual relationships (e.g. 
the owners of nursing homes, or retirement homes). The different motives are not necessarily 
all sharply distinguishable. There may be fuzzy intermediate cases, including mixed motives. 
I have mentioned motives because they are important when considering the ethics of 
selection and use of ACs, but will not discuss them further. 


Problems of achieving the enabling functions 
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I work on robotics, not in order to produce useful machines, but because that is a very good 
way of addressing many old philosophical problems (Sloman, 1978), and also because it is a 
good way to attempt to understand some of the problems solved by biological evolution. It is 
also possible for such research to produce practically useful designs. However, I have no 
interest in making or using machines with the “engaging” functions described previously, i.e. 
those described earlier as: toys, engagers and dummy-humans. I have no objection to others 
building these things - for the right motives, though I don’t think I would ever want to use 
them myself. E.g. I intensely dislike pseudo-human interfaces, with smiling, nodding, 
expressions, moving eyes, “emotional” voices, and a stock of linguistic stratagems for 
expressing concern, interest, or understanding, whether what is expressed exists or not. I hope 
nobody ever expects me to put up with such things if I become disabled enough to need a 
AC. The widespread annoyance produced by Microsoft’s “helpful” paperclip suggests that I 
am not alone. Unfortunately, researchers, especially if they are engineers rather than 
experienced psychologists, often assume for bad reasons that it is obvious that certain kinds 
of interface will be preferred by users. 


The rest of this chapter is concerned only with a subset of the problems of producing ACs 
with enabling functions. Really useful general-purpose ACs are very difficult and way out of 
reach in the foreseeable future, for reasons that do not seem to be widely understood and 
which I shall try to explain. The arguments are not the same as those used by opponents of AI 
e.g. (Dreyfus, 1979, Searle, 1980, Weizenbaum, 1976): I am not claiming that it is impossible 
to produce human-like machines, or that human-like machines cannot be based on digital 
computers. I am claiming that the products of evolution and individual human development 
include many competences that have a depth and complexity that needs to be understood 
much better if we are to produce really good digital Companions of the enabling type. It will 
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be much harder to achieve the enabling functions in a machine than most people realise, 
because it will be necessary to provide some of those competences that are not yet 
understood. The currently most popular AI techniques for competent machines, making 
heavy use of data-mining and statistical learning, are inadequate for the task. 


The main point is that the detailed requirements for ACs to meet the enabling/helping 
specifications are not at all obvious, and have implications that make the design task very 
difficult in ways that have not generally been noticed. Building such systems will require a 
deep new understanding of some hitherto unexplained human competences. Perhaps they will 
eventually be achieved if we analyse the problems properly. 


Kitchen mishaps 

Many of the things that crop up will concern physical objects and physical problems. 
Someone I know knocked over a nearly full coffee filter close to the base of a cordless kettle. 
This caused the residual current device in the fuse box under the stairs to trip, removing 
power from many devices in the house. Fortunately she knew what to do, unplugged the 
kettle and quickly restored the power. However, she was not sure whether it was safe to use 
the kettle after draining the base, and when she tried it later the RCD tripped again, leaving 
her wondering whether it would ever be safe to try again, or whether she should buy a new 
kettle. In fact it proved possible to open the base, dry it thoroughly, then use it as before. 


Should a AC be able to give helpful advice in such a situation? Would linguistic interaction 
suffice? How? Will cameras and visual capabilities be provided? People who work on 
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language understanding often wrongly assume that providing 3-D visual capabilities will be 
easier, whereas very little progress has been made in understanding and simulating human¬ 
like 3-D vision and spatial understanding, which involves far, far more than recognising 
things. Human vision includes a wide and deep collection of competences, and ontologies, 
some analysed in (Sloman, 2008a). Many researchers confuse seeing with recognising, which 
is wrong because you can see something, and even interact with it, without recognising it, e.g. 
by pushing it out of your way or smashing it with a sledgehammer. Moreover, even when 
something is recognised either as a type of entity or, as a previously encountered individual, 
there is more to seeing than that. For instance, as Gibson pointed out, seeing and other forms 
of perception provide information about positive and negative affordances, and such 
perception usually builds on the more basic competence of perceiving which processes are 
occurring or can occur in a situation. As argued in (Sloman, 2008a), that requires the use of 
forms of representation that can encode information about processes occurring in the 
environment, including processes that involve multiple concurrent changes, e.g. of shape, 
position, orientation, and relationships between objects, some metrical, some semi-metrical 67 
some topological, some functional, and some causal. It is also useful in many contexts to 
understand the behaviors of different kinds of material, e.g. fur, skin, porridge, metal, plastic, 
wood, water, detergent, powders of various sorts, paper, cloth, cardboard, cotton wool, etc., 
and at present no machines come close. As far as I know the problem of giving machines an 
understanding of how shape and materials interact in the production of behaviors has barely 
been recognized, let alone solved. 


There are no machines that have such capabilities, partly because only a very narrowly 
restricted set of requirements for vision has been studied by researchers in vision and 
robotics, and partly because it is very hard to produce appropriate designs, for reasons 
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developed in more detail in (Sloman, 2008a). Moreover some of the fonns of representation 
and processing capabilities are needed for thinking about spatial structures and processes 
even when they are not currently perceived, for instance, when planning actions, or working 
out what an opponent might do but is not yet doing. 


For these reasons, it is likely that many of the functions of a AC that can “advise, comfort and 
carry out a wide range of functions to support diverse personal and social needs, such as to be 
artificial Companions for the elderly, to help their owners leam or to sustain the fitness and 
health of their owners” will require them to have access to cameras and other sensors and to 
be able to reason about practical problems involving spatial structures and processes. It is not 
always noticed that these capabilities are required even for purely linguistic interactions. E.g. 
the person who spilt coffee over a kettle base told me what had happened over the phone and 
we were able to discuss possible strategies because I could think and reason about the 
physical configurations involved without actually seeing them. When I later, out of curiosity, 
attempted to use Google to find advice about what to do in such a situation, the closest things 
I found were advertisements for kettles with sealed bases that were protected against spills. 


It is not always appreciated how much humans learn about space, time, structures, processes 
and causal interactions in the first few years of life, providing a deep and general 
understanding of many features of the environment on which a wide range of later 
developments can build. If ACs, whether robots or not, are to emulate the depth of 
understanding, creativity and flexibility that can be based on those spatial competences we 
shall need far more progress in designs suited for use in robots which can also be used in 
disembodied ACs. 


293 



It is also not widely appreciated that recent trends in AI and cognitive science that disparage 
symbolic AI and emphasise the role of embodiment and the dynamical interactions between 
bodies and the environment, e.g. (Brooks, 1990, 1991) leave out of account many human 
cognitive functions that involve thinking about, reasoning about and planning actions in an 
environment, which require forms of representation and architectures that are different from 
those needed for online interaction, as explained in (Sloman, 2009, 2008c). 


Identifying affordances and searching for things that provide them 

Understanding a human need and seeing what is and is not relevant to meeting it may require 
creative recombination of prior knowledge and competences. 


Suppose an elderly user finds it difficult to keep his balance in the shower when soaping his 
feet. He prefers taking showers to taking baths, partly because showers are cheaper. How 
should the AC react to this problem? Should it argue for the benefits of baths? Should it 
send out a query to its central knowledge base asking how people should keep their balance 
when washing their feet? (It might get a pointer to a school for trapeze artists, or a balancing 
pole suitable for tight-rope walkers.) 


What if the AC designer had not anticipated that problem, and nothing suitable was in the 
latest knowledge updates for the AC? What are the requirements for the AC to be able to 
invent the idea of a folding seat attached to the wall of the shower that can be temporarily 
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lowered to enable feet to be washed safely in a sitting position? Alternatively what are the 
requirements for it to be able to pose a suitable query to a search engine? How will it know 
that safety harnesses and handrails are not good solutions? Of course, once it had understood 
that a folding shower seat might be a good solution, it might use that to direct searches for 
local suppliers of shower seats. If it found none it might have to dream up an alternative 
solution so as to be able to decide what to search for on the internet. There are many more 
examples of issues that can arise in a domestic situation where individuals of various sorts 
need help and advice, requiring the helper to think about interacting 3-D structures and 
processes. 


Another example: I expect very few people reading this have ever spilt a bowl of porridge on 
a carpet. Yet you probably know quite a lot about carpets in general (or a particular type of 
carpet, e.g. cord, mohair, wilton, etc.), about physical qualities of porridge, and about 
cleaning messes, and you can combine what you know to give to give specific advice about a 
novel situation. For example that will rule out certain approaches, such as using a vacuum 
cleaner, a broom, a bucket of water and a mop. Ruth Aylett responded to the example by 
pointing out that someone carrying a bowl of porridge may also be carrying a spoon, which 
would be one of the best devices to use, in conjunction with the bowl, to start cleaning up the 
mess. 


A AC could be pre-programmed with many specific pieces of advice to offer in specific 
situations, but providing the ability to deal creatively with novel problems by combining 
existing competences and knowledge in a novel way, is much harder, and requires the AC to 
have a much deeper understanding of space, time, causation, and interactions between 
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different types of matter and the various shapes in which they can occur and how the 
combination of matter and shape can influence the processes that are possible. 


Remembering particularities: episodic memory 

Another important human competence that no AI system yet has included is the construction 
of so-called “episodic memories”, spanning different regions of space and time at different 
scales. This involves learning not only new concepts and new generalisations, but also 
acquiring and storing information about particular places and times, and at least sometimes 
being able to recall such particularities when they are relevant to a new problem. For 
example, as you move around a room, or various parts of a building, or a town, you can at 
any time see only a small spatial region, yet you remain aware that that there are things you 
have previously seen or read about or been told about that exist but are out of sight, and you 
are also aware of things you have done and other happenings that occurred recently, and 
which no longer have any impact on the senses. Some robots that can do SLAM, i.e. 
Simultaneous Localisation and Mapping, have a special case of that competence, though 
usually the infonnation stored, and what can be done with it, is very limited. 


This infonnation about what exists unsensed now, or existed in the past, or may exist in the 
future, has many applications, including planning routes and actions, describing the 
environment or recent events to others, and also understanding what other speakers are 
referring to when they talk about the things or past happenings in the environment. So these 
spatial competences play an essential role in human linguistic competences, of kinds that the 
AC is likely to need. 
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Giving machines an understanding of physical and geometrical shapes, processes and causal 
interactions of kinds that can occur in an ordinary house is currently far beyond the state of 
the art. Compare the ‘Robocup@Home’ challenge, still in its infancy 
(http://www.ai.rug.nl/robocupathome/). Major breakthroughs of unforeseen kinds will be 
required for progress to be made, especially breakthroughs in vision and understanding of 3- 
D spatial structures and processes. Some simple examples of the required competences are 
discussed in an online discussion paper on “Predicting Affordance Changes”. 68 Of course, 
one response to all this would be to aim for much simpler and more easily attainable 
competences, such as simply providing the engaging functions. Either that, or implementing 
fairly simple helper packages is probably what many providers of ACs will aim for in the 
near future. I shall ignore such goals in the rest of this chapter and focus on requirements for 
addressing the harder problems. 


More abstract problems 

So far all the problems presented relate to a richly varied 3-D environment in which 
structures and processes occur that need to be understood, enabled, prevented, predicted, 
explained, coped with, undone, and so on, which have so far defeated AI systems except for 
very special, simple cases, although many of the required competences are found in young 
children and non-human animals. 


There are also meta-semantic competences that young humans develop, namely the ability to 
represent some things, including themselves, as infonnation processors, and to develop and 
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use explicit or implicit theories about states and processes that can occur in information 
processors, such as having and using concepts and being able to understand some things 
while failing to understand others. Some meta-semantic competences involve 3-D structures 
and processes, for instance the ability to infer the intention in an observed action, but not all 
do, for instance, wondering whether someone understands long division. Many, though not 
all, uses of meta-semantic competences involve using language to describe mental states and 
processes, of oneself or others. Although the problem of getting a machine to handle 
referential opacity, the fact that an Understander will not in general know whether 
descriptions refer to the same object or not, has been recognised for a long time (e.g. see 
McCarthy and Hayes, 1969) there is no agreement on how to solve the problem. Some 
researchers favour using special kinds of modal logic, or other notational devices. I suspect 
that the best solution is to use an infonnation-processing architecture that supports 
encapsulation, e.g. an architecture that allows beliefs and inferences of other individuals to be 
simulated, which need not involve treating accepting them as one’s own. 


Coping with referential opacity is one among many problems that arise when using language 
to communicate. Another competence that interfaces with the ability to acquire and store 
information about the physical environment, including portions that are not in view, and 
events that occurred in the past, is the ability to understand referring expressions, including 
indexicals, proper names and definite descriptions (e.g. “The mug I used yesterday”). 
Machines that cannot build human-like information structures referring to the spatial and 
temporal environment including currently unobservable portions will not be able to resolve 
ambiguities and interpret all referring expressions. 
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Is training the solution? 


Many AI researchers have addressed problems of communication by machines, using natural 
language (written, spoken, or encoded in other media), gestures, facial expressions, bodily 
postures, etc. However, much of this work has involved either providing shallow behavioral 
rules directly, or else enabling machines to be trained to produce different responses to 
different situations by being presented with large numbers of examples. 


This may work for simple fonns of communication in a restricted range of contexts, but for 
more general human-like understanding of behaviors and communications that are not all 
instances of a fixed class of stereotypes, the understander needs a richer, deeper theory about 
the kind of thing a communicator can be, including a theory about the information processing 
architecture that lies behind expressions of beliefs, intentions, desires, curiosity, amusement, 
interest, boredom, disgust, pleasure, regret, shame, embarrassment, annoyance, indignation, 
surprise, puzzlement, wanting to understand, misunderstanding, incomprehension, refusal, 
acceptance, willingness, resigned acquiescence, suspicion, attending, noticing, ignoring, 
expecting, remembering, forgetting, preference, mood, weariness, jauntiness, alertness, 
sleepiness, being intoxicated, and many more. 


Beyond behavioral dispositions 

These states and processes may or may not manifest themselves directly in behavior, but they 
are not pure behavioral dispositions, since they depend on internal structures and processes in 
a rich and complex infonnation processing architecture, including an architecture that 
supports meta-semantic competences used both in self-understanding and in other- 
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understanding. The behaviors are the outcome of interactions between components in the 
architecture. We still do not understand precisely what variety of architectures humans can 
have, and what sorts of implicit theories individuals develop about their own and others’ 
architectures that enable them to get on in life with only partial understanding of the 
enonnous complexity of other people, though there are many disciplines investigating this 
from different viewpoints, using different ontologies and methodologies, including 
conceptual analysis in philosophy (Ryle, 1949). 


Much research in AI ignores relevant work in other disciplines, including philosophy, and 
because much of the work is done by people trained in electronics, engineering, computer 
science, mathematics, etc. and not in linguistics, psychology, or philosophy, this leads to 
over-simplified models. There are attempts to explore more of the complexity required, e.g. 
(Minsky, 2006), but the work is still at a very early stage. We therefore cannot expect ACs in 
the next few decades to be able to do more than produce shallow responses on the basis of 
shallow understanding of what they perceive. 


This may work well enough in situations where the human-machine interaction is constrained 
to a very narrow task, such as booking train tickets, asking directions in a department store, 
filling in an order form, enquiring about legal rights or a certain medical condition, for 
example. But the situations envisaged for ACs as described in the invitation to contribute this 
paper will not be so constrained, since ACs will have to go into different homes, with 
different sorts of people whose lives, loves, memories, capacities, incapacities and 
requirements can vary enormously, and also change over time. 
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The Examples given previously illustrate the need for creative understanding of physical 
problems. There will also be a need for creative understanding of thought processes, 
cognitive competences, preferences, motivations and values. There is space here only for a 
few examples. Suppose the user U has decided to compile a database of information about his 
relatives and close friends and asks the AC to help with the task. Suppose U is an intense 
atheist, and while trawling for information about U’s siblings the AC finds that U’s brother 
has written a blog entry supporting creative design theory, or discovers that one of U’s old 
friends has been converted to Islam and is training to be a Mullah. How should the AC react? 
Compare discovering that the sibling has written a blog entry recommending a new detective 
novel he has read, or discovering that the old friend is taking classes in cookery. What about 
getting evidence suggesting that U’s spouse had been having an affair with a friend long ago? 
Could the AC reason that telling U about it might assuage guilt about an affair U had once 
had? 


A AC may need the ability to take account of emotional responses that news items may 
produce. How will it work out when to be careful when reporting new information? Where 
will the ACs goals, preferences, and values come from? 


Is the solution statistical? 
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The current dominant approach to developing language understanders and advice givers 
involves mining large corpora using sophisticated statistical pattern extraction and matching. 
This is much easier than trying to develop a structure-based understander and reasoner, and 
can give superficially successful results, depending on the size and variety of the corpus and 
the variety of tests. But the method is inherently broken because as sentences get longer, or 
semantic structures get more complex, or physical situations get more complex, the 
probability of encountering recorded examples close to them falls very quickly. Then a helper 
must use deep general knowledge to solve a novel problem creatively, often using non- 
linguistic context to interpret many of the linguistic constructs. Some machines can already 
do creative reasoning in restricted domains, e.g. planning and mathematical reasoning, but 
they are still very limited. 


Why do statistics-based approaches work at all? 

The behavior of any intelligent system, or collection of individuals, will leave traces that may 
have re-usable features, and the larger the set the more re-usable items it is likely to contain - 
up to a point. For instance it may not provide items relevant to new technological, or cultural 
developments or to highly improbable but perfectly possible physical configurations and 
processes. So any such collection of traces will have limited uses, and going beyond those 
uses will require something like the power of the system that generated the original 
behaviors. 


In humans (and some other animals), there are skills that make use of deep generative 
competences whose application requires relatively slow, creative, problem solving, e.g. 
planning routes. But practice in using such a competence can train powerful associative 
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learning mechanisms that compile and store many partial solutions matched to specific 
contexts (environment and goals). As that store of partial solutions (traces of past structure- 
creation) grows, it covers more everyday applications of the competence, and allows fast and 
fluent responses. 


Some statistical AI systems that cannot generate the data can infer those partial solutions 
from large amounts of data. But because the result is just a collection of partial solutions it 
will always have severely bounded applicability compared with humans, and will not be 
extendable in the way human competences are, namely by creatively recombining things 
known previously so that solutions to problems can be derived from the combination, as in 
designing a new type of machine made of familiar parts. Moreover, if trained only on text a 
machine will have no comprehension of non-linguistic context. Dealing with novel problems 
and situations requires different mechanisms that support creative development of novel 
solutions. 


If the deeper, more general (and slower) competence is not available, wrong extrapolations 
can be made, inappropriate matches will not be recognised, new situations cannot be dealt 
with properly and further learning will be very limited, or at least very slow. In humans the 
two systems work together to provide a combination of fluency and generality, not just in 
linguistic competence, but in many other domains. 


Occasionally I meet students who manage to impress some of their tutors because they have 
leamt masses of shallow, brittle, superficially correct patterns that they can string together - 
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without understanding what they are saying. They function like corpus-based AI systems, and 
are not much good as (academic) Companions. 


Can it be done? 

Producing a AC of the desired type may not be impossible, but is much harder than most 
people realise and cannot be achieved by currently available learning mechanisms. Solving 
the problems will include: 

1. Learning more about the forms of representation and the knowledge, competences 
and meta-competences present in prelinguistic children who can interact in rich and 
productive ways with many aspects of their physical and social environment, thereby 
continually learning more about the environment, including substantively extending 
their ontologies. Since some of the competences are shared with other animals they 
cannot depend on human language, though human language depends on them. 
However we know very little about those mechanisms and are still far from being able 
to implement them. 

2. When we know what component competences and forms of representation are 
required, and what sorts of biological and artificial mechanisms can support them, we 
shall also have to devise a self-extending architecture which combines them all and 
allows them to interact with each other, and with the environment in many different 
ways, including ways that produce growth and development of the whole system, and 
also including sources of motivation that are appropriate for a system that can take 
initiatives in social interactions. No suggestions I have seen for architectures for 
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intelligent agents, come close to requirements for this, though there are many AI 
projects taking small steps that could be useful. 


I suggest the only reliable way to meet the objectives is to understand and replicate, and later 
on to build on, some of the generic capabilities of a typical young human child, including the 
ability to want to help. 

What is needed? 

Before human toddlers leam to talk they have already acquired deep, reusable structural 
information about their environment and about how people work. They cannot talk but they 
can see, plan, be puzzled, want things, and act purposefully. In the first two or three years of 
life they are constantly learning about large numbers of affordances related to various kinds 
of processes involving objects and situations in the environment. They are also learning about 
epistemic affordances - learning to discern situations that do and do not provide good task- 
specific information. 


They have something to communicate about. That pre-linguistic competence grows faster 
with the aid of language, but must be based on a prior, internal, formal ‘linguistic’ 
competence using fonns of representation with structural variability and (context-sensitive) 
compositional semantics. Jackie Chappell and I have begun to call these forms of 
representation, which can occur either inside the head or in some public communication, 
“Generalised languages” or “GL"s in (Sloman & Chappell, 2007, Sloman, 2008b). 
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The use of GLs enables very young children to learn any human language and to develop in 
many cultures. ACs without a similar pre-communicative basis for their communicative 
competences are likely to remain shallow, brittle and dependent on pre-leamt patterns or rules 
for every task. 


Perhaps, like humans (and some other species), they can escape these limitations if they start 
with a partly ‘genetically’ detennined collection of meta-competences that continually drive 
the acquisition of new competences building on previous knowledge and previous 
competences: a process that continues throughout life. The biologically general mechanisms 
that enable humans to grow up in a very wide variety of environments, are part of what 
enable us to leam about, think about, and deal with novel situations throughout life. Very 
little is understood about these processes, whether by neuroscientists, developmental 
psychologists or AI researchers, and major new advances are needed in our understanding of 
information-processing mechanisms. Some pointers towards future solutions are in the online 
presentations listed below. 


Alternatives to canned responses 

The kitchen mishap, shower support problem, and spilt porridge problem were just three 
examples among a vast array of possibilities that a domestic AC may need to help with. Of 
course, if the designer anticipates such accidents and problems, the AC will be able to ask a 
few questions and spew out relevant canned advice, and even diagrams showing how to open 
and dry out the flooded base. 
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But suppose designers had not had that foresight: What would enable the AC to give sensible 
advice? If the AC knew about electricity and was able to visualise the consequences of liquid 
pouring over the kettle base, it might be able to use a mixture of geometric and logical 
reasoning creatively to reach the right conclusions. It would need to know about and be able 
to reason about spatial structures and the behavior of liquids. Although Pat Hayes described 
the ‘Naive physics’ project decades ago, it has proved extremely difficult to give machines 
the kind of intuitive understanding required for creative problem-solving in novel physical 
situations. In part that is because we do not yet understand the forms of representation 
humans (and other animals) use for that sort of reasoning. 


A AC lacking similar mechanisms and a similar deep understanding of our environment may 
cope over a range of circumstances that it has been trained or programmed to cope with - and 
then fail catastrophically in some novel situation. Can we take the risk? Would you trust 
your child with one? 


Conclusion 

A lot of research is being done in connection with the long tenn goal of developing digital 
Companions, some of which may provide important enhancements for the quality of life 
especially in a population with ever increasing life-expectancy that does not always go with 
full physical and mental health in later years. However, there are risks in this research. One 
such risk is exaggerating the likely value of some of the easier research based on producing 
shallow behavioral competences in robots and non-robotic systems. The fact that the design 
goals of such projects are met carries no guarantee that the design goals are worth pursuing. 
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Many of them will be of some use in providing entertainment and fun, the “engaging 
functions” of ACs, without necessarily providing adequate “enabling functions”. 


Another risk is underestimating the difficulties of producing digital Companions that are able 
to provide the enabling functions, i.e. able to be helpers or even caring and developing 
helpers. Progress in providing those functions will require progress in meeting more general 
goals of AI research, including some goals that are required for producing useful general 
purpose domestic robots. I argued above that despite the early promise of designs based on 
use of statistical methods applied to large training sets, such designs are inherently limited by 
the “training envelope” and will not be capable of creative problem solving in dealing with 
novel situations. 


That second difficulty can be tackled by doing far more analysis of the nature of the 
environment than is usual in AI research, which tends to focus more on mechanisms and 
architectures, on the assumption that good designs will work in any environment. It seems 
likely however, for reasons I have tried to illustrate without giving any proofs, that progress 
in this area will require us to understand what sort of ontology an AC needs to start with, how 
that ontology can develop over time, including changing in response to changes in the 
environment, along with developing good fonns of representation and architectures to allow 
information using such ontologies to be acquired, stored, modified, generalised, de-bugged, 
and used creatively in solving new problems. In particular, this will require machines to be 
given much better ways of representing processes that can occur in the environment including 
processes involving interactions between parts of objects, and different kinds of material. 
Contrary to current popular views this does not require the AC to share a human body 
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morphology, though it does require information processing of kinds that occur in human 
minds, only a small subset of which is concerned with direct interaction with the 
environment. The topic is large and complex, and I have given pointers to online 
presentations and papers exploring the issues in more detail. 


Ethical issues 

There are several ethical issues related to the design, selection, and use of ACs. 


Although many researchers assume that ACs will simply be machines that have no more 
ethical status than any other domestic appliance, it is arguable for reasons given in above that 
an AC caring for one or more individuals over a long time period in which many things 
change, will need to be able to have its own desires, including wanting to help the users, if it 
is to be successful caring Companion, including meeting needs that were not anticipated 
when the AC was originally designed, and including dealing with cases where there are 
conflicts between what the user claims to want and what would be in the user’s best interests. 


If providing effective Companionship requires intelligent machines to be able to develop their 
own goals, values, preferences, attachments etc., including really wanting to help and please 
their owners, then if some of them develop in ways we don’t intend, will they not have the 
right to have their desires considered, in the same way as children do if they develop in ways 
their parents don’t intend? This was discussed briefly in the epilogue to (Sloman, 1978). 69 It 
can also be argued that Asimov’s laws of robotics are immoral, because they are unfair to 
future robots which may have their own preferences, desires and values. 70 The cavalier 
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suggestion that the interests of future machines, no matter how sophisticated they become, 
can always be disregarded, or at least given lower priority than the interests of human beings 
seems to indicate an assumption that humans are necessarily uniquely important among 
information processing systems, an assumption of human ethical centrality that should, in the 
long run, go the way of the assumption that humans inhabit the centre of the physical 
universe. 


You Really need to know what your Bot(s) are Thinking about You - Alan 
FT Winfield 


Abstract 


The projected ubiquity of personal Companion robots raises a range of interesting but also 
challenging questions. There can be little doubt that an effective artificial Companion, 
whether embodied or not, will need to be both sensitive to the emotional state of its human 
partner and be able to respond sensitively. It will, in other words, need artificial theory of 
mind - such an artificial Companion would need to behave as if it has feelings and as if it 
understands how its human partner is feeling. This chapter explores the implementation and 
implications of artificial theory of mind, and raises concerns over the asymmetry between and 
artificial Companion's theory of mind for its human partner and the human's theory of mind 
for his or her artificial Companion. The essay argues that social learning (imitation) is an 
additional requirement of artificial Companion robots, then goes on to develop the idea that 
an artificial Companion robot will not be one robot but several. A surprising consequence of 
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these ideas is that a family of artificial Companion robots could acquire an artificial culture of 
its own, and the essay concludes by speculating on what this might mean for human(s) 
interacting with their artificial Companion robots. 


Introduction 


In this volume, Peltu and Wilks set out the potentially desirable features of an artificial 
Companion in response to the question: what features are likely to make a 'good 
Companion’? The same paper then refines this feature set by addressing the question: how 
human should an artificial Companion (AC) attempt to be. A clear conclusion is that an 
effective AC will need to be endowed with the capacity to make an appropriate emotional 
response, that we can label artificial empathy, and with a built-in “...autonomous modelling 
capability to enable the AC to generate its own behavior according to its perception of the 
interaction with the user and the environment”. In the context of an artificial Companion this 
autonomous modelling capability is, in effect, a theory of mind. 


This essay considers the implications of artificial empathy and theory of mind in embodied 
Artificial Companions, in other words Robots. With the caveat that meeting the potentially 
desirable feature set in [Peltu & Wilks] is technically well beyond the current capability of 
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robotics, I first consider how we might implement artificial theory of mind - and hence 
artificial empathy - in a robot; and what this might imply in terms of the two kinds of minds 
(human and artificial) in the relationship between a human and his or her artificial 
Companion. 


The essay then goes on to consider the implications of embodiment and what additional 
constraints and/or requirements follow. In particular we consider firstly, social learning, and 
secondly the possibility that an embodied AC might in fact need not one but several bodies. 
Our embodied artificial Companion might in reality turn out to be not one but a small family 
of autonomous robots (some mobile, some not), able to leam not only from their human 
charges but from each other. The essay concludes with a cautionary discussion about what 
this might mean for the humans in this novel relationship. 


How an Artificial Companion might have Theory of Mind 


Theory of mind is the term given by philosophers and psychologists for the ability to form a 
predictive model of others [Car 1996]. With theory of mind, it is supposed, we are able to 
understand how others might behave in particular circumstances. We can empathise because 
theory of mind allows us to imagine ourselves in the situation of others and hence feel as they 
feel. Damaged or impaired theory of mind has been has been conjectured as giving rise to 
autism in humans [Hob 1990]. However, the idea of theory of mind is empirically weak - we 
know it should exist but we have only a poor understanding of the neurological or cognitive 
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processes that make theory of mind. Artificial Intelligence (AI), and its embodied counterpart 
- robotics, provide an interesting constructionist approach to theory of mind because they 
allow us to ask the question “how would we build artificial theory of mind in a robot?”. In the 
emerging field of machine consciousness, for instance, Holland and Goodman argue that if 
we put a computer simulation of both the world and another robot in the artificial brain of a 
robot, then it has the ability to ask “what if’ questions about how the other robot might 
behave by running simulations [Hol2003]. Since computer simulation is a well developed 
technology in robotics then Holland and Goodman's approach offers an apparently 
straightforward route to implementing a simple kind of behavioral artificial theory of mind in 
robots. Furthermore, the approach offers the possibility that a robot's theory of mind might 
improve over time. Consider two robots A and B, where A has a simulation of B in its 
artificial brain. A could run several simulations of B before deciding what to do next. A could 
then observe how B actually behaves in response and compare that observation with how it 
'thought' B would behave (in simulation). The difference between the two could then be used 
to modify A’s internal model of B. In this way A improves its artificial theory of mind, for B. 


This discussion has focussed on how a robot could have an artificial theory of mind for 
another robot (or indeed, for itself), but in principle there is no reason to suppose that same 
approach couldn’t be used for a robot to have (or learn) theory of mind for any other agent 
with which it interacts, including humans. Now one could argue that it is one thing for a robot 
to have theory of mind for another robot, with its relatively simple repertoire of behaviors, 
but quite another for a robot to have theory of mind for a human being with all of the nuanced 
complexity of human behavior. But an effective digital Companion surely must have some 
theory of mind in order to be able to demonstrate artificial empathy (or a workable analogue). 
It is possible therefore for a robot to be imbued with some simple theory of mind for a human 
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being which it can then develop and improve with experience. Models of human emotional 
responses are being actively explored in the field of research known in Japan as 'Kansei', see 
for instance [Suzl998]. 


Consider now the relationship between a human and his or her artificial Companion robot. If 
we accept the argument above then the robot can have a theory of mind for the human and, 
furthermore, that theory of mind can improve over time. But what kind of theory of mind 
does the human have for its robot Companion? There is an asymmetry here, because the 
robot's theory of mind has been carefully engineered with, as a starting point, a model of 
human behavior and psychology (with some level of fidelity), and the ability to learn from 
experience. In contrast the human has no theory of mind for its robot Companion. Of course, 
humans have an extraordinary propensity to anthropomorphize so will quickly assume some 
theory of mind for their robot Companions. But that theory of mind is almost certain to be 
wrong. The question then is: does it matter if we have the wrong theory of mind for our AC 
robots? I believe that it does. Because our AC robot has been designed with both 
intentionality, or at least to behave as if it has intentionality, and with artificial empathy, then 
when it acts or reacts in a way that we didn’t expect, we will be much more disconcerted than 
we are when machines without agency behave in ways we do not expect. My point is this: if 
your car or washing machine behaves in a way you do not expect, then you fix it without 
worrying about whether you might hurt its feelings; if your AC robot behaves unexpectedly 
do you (a) fix it, (b) interact with it in an effort to get it to change its behavior (ask it nicely to 
adjust its theory of mind), or (c) modify your expectations (adjust your theory of mind for it 
and avoid hurting its artificial feelings). 
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Artificial Companion robots will need to be able to learn socially, by imitation 


I above argued that an AC robot with theory of mind might be able to improve its theory of 
mind by, in effect, learning from experience. Our AC robot would be classified as a 
Popperian Creature within Dennett’s Tower of Generate-and-Test, because it has the ability to 
internalise and test (in its simulated model of other robots or humans) possible actions before 
trying them out [Denl995]. With the ability to modify its internal model our AC robot can 
individually learn from its own experience. 


I now argue that an AC robot needs to be able to leam socially (i.e. by imitation), as well as 
individually, for the following reasons. Firstly, an embodied AC will be embodied precisely 
because it needs to physically interact with its human charges and its environment. That 
physical interaction is likely to be by means of manipulators (arms and hands with fine motor 
skills) coupled with mobility (legs or wheels), and an AC robot is highly likely to be 
humanoid because it needs to share human living or work spaces and, quite possibly, use 
human tools and appliances. Given that human spaces are chaotic and constantly changing 
our AC robot cannot possibly be pre-programmed with all possible physical interactions - 
instead it will have to leam those interactions. By far the easiest and most natural way to 
leam many such interactions will be by observing and imitating a human. In other words a 
human should be able to teach an AC robot how to perfonn a task, by simply perfonning the 
task. 
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The second reason that I believe the ability to learn socially (by imitation) to be an essential 
feature of an AC robot goes far beyond utility, or naturalness. Dautenhahn has argued that 
“social intelligence for artefacts ... may be a necessary prerequisite for those scenarios in 
which autonomous robots are integrated into human societies, interacting and communicating 
both with humans and with each other” [Daul995]. Researchers within the emerging 
interdisciplinary field of imitation and social learning in robotics, humans and animals 
[Neh2007] suggest that imitation and theory of mind are strongly interdependent. In that 
volume, for instance, Williams writes of the imitative origins of mind-reading [Wil2007]. I 
believe that this body of work strongly suggests that an AC robot will be a more useful and 
effective Companion if its theory of mind incorporates social learning. 


An Artificial Companion robot will not be one robot but several 


When we think of digital Companion robots we think of one robot as a Companion to one 
human, or perhaps one household. The reality is likely to be more complex. Because all the 
functions of our robot are unlikely to be embodied in a single machine, it's likely that instead 
of a single complex multi-functional Companion robot we will have several simpler single- 
or few-function specialist robots. For instance the robot that vacuums the floor is highly 
unlikely to be the same robot that provides healthcare monitoring and assistance; the 'butler' 
robot of the movie Bicentennial Man is, I believe, highly unlikely. Of course it is then not 
only likely but inevitable that the several robots that, collectively, comprise our digital 
Companion are networked and thus wirelessly able to share information to coordinate their 
actions or alert each other to the needs of their human(s). 
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This raises the important question: is it better to design such a system as several robots each 
able to perform its specialist function autonomously and independently of the others but also 
able to communicate with each other if necessary, or is a single centralised Artificial 
Intelligence (perhaps with the theory of mind discussed above) able to control and coordinate 
a number of specialist but ’dumb’ physical robots better? The first approach we can think of 
as a 'society' of embodied robots cooperating to meet our needs, the second as a single digital 
Companion with a ’distributed’ body. 

I argue that the first approach: a society of cooperating robots is both more likely and much 
more desirable. More likely for pragmatic reasons: specialist function robots are significantly 
easier to design than complex multi-functional ones, and more desirable for several reasons: 
Firstly reliability; if one robot fails the others will be unaffected in their ability to continue 
their specialist functions; secondly, flexibility, because you only need the set of robots that 
meet your particular needs. More importantly, I would argue that a society - or perhaps 
family might be a better word - of robots that collectively meet your Companionship needs 
will be more socially acceptable than a single AI with multiple robot bodies. I believe the 
latter approach could be deeply unsettling because the multiple robot bodies of the centralised 
AI will appear to speak with the same voice, and that voice will be perceived as a 
disembodied all-seeing centralised AI (cf HAL in the movie 2001). In contrast, the family of 
robots approach will be comforting since each autonomous robot will have a different 
’personality’ - like a family of pets. Why will they each have a different personality? Because, 
although each robot may well be programmed with the same ’initialised’ theory of mind, each 
will quickly acquire a different learned experience of interacting with its human charges and 
the world, through its unique body, and those learned differences will be manifest as different 
personalities. 
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A family of Artificial Companion Robots could acquire an Artificial Culture of its own 


This essay has argued that an AC robot will need theory of mind, and the ability to learn both 
individually through experience, and socially by imitation. And furthermore, that an AC 
robot will not be one robot, but several wirelessly networked, each with specialised physical 
functions or capabilities. It follows that each of the robots in this 'family' that collectively 
comprises the AC will, subject to their physical differences and limitations, be able to imitate, 
and hence socially leam, from each other, as well as their human Companions. If we adopt 
the terminology of memetics, and accept the notion that memes 71 copied by imitation from 
one robot to another may be subject to the three evolutionary operators of variation, selection 
and heredity, then it follows that new patterns of socially learned behavior could emerge and 
(memetically) evolve within a family of AC robots [Blal999]. These patterns of behavior 
would represent an emerging artificial culture. 


Of course memetic transmission could (and indeed will) occur between humans and robots 
but, because this will be mediated largely by humans teaching their AC robots it is, I would 
suggest, unlikely to result in a free-running emerging human-robot culture. The asymmetry 
discussed earlier would hinder any emerging human-robot culture. More likely, I believe, is 
that a robot-robot culture will emerge and evolve within the AC robots; the fact that the 
robots which collectively comprise the AC share the same basic AI and are - in effect 
telepathically - linked will allow the artificial culture to evolve very quickly. This emerging 
robot-robot artificial culture will of course be inscrutable to humans; it will be, in essence, an 
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alien or exo-culture. As Blackmore writes in [Bla2003] if AC robots' “...perceptual and 
categorisation systems were different from ours then their language might be extremely 
difficult for us to leam. We would find it hard to understand what they were talking about 
because they would parse the world in different ways and talk about completely different 
things from us”. 


Concluding discussion 


What might the arguments presented in this chapter suggest about how we might approach 
the future design, or practice, of robots as artificial Companions? This essay has, in essence, 
raised questions and concerns about the asymmetry of kinds of minds: human and artificial, 
both at an individual level and - significantly amplified - at the cultural level. 


We can conclude that at an individual level it is likely that a relationship between a human 
and his or her artificial Companion will need to be one of compromise. Thus, even though 
our AC robot is an artefact, we will need to be prepared to learn its ways, just as it will have 
to learn our ways. This, I believe, is significantly and properly different to the common 
expectation of robots as servants, or even slaves. 


At a cultural level the problem is potentially more challenging. We could mitigate the 
possibility of an emerging and evolving robot culture inscrutable, and therefore 
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disconcerting, to humans 72 . By engineering the AC robots perceptual and categorisation 
systems to be just like ours [Bla2003]. But this will be difficult, perhaps even impossible. Our 
artificial Companions will see and experience the world very differently to us, not least 
because their sensory systems will be different from ours, even if we were to tune our AC 
robots' sensors to the same range of colours and sounds that we experience. We might need 
instead to embrace and celebrate a new kind of robot culture complementing and enriching 
our own. A dress rehearsal perhaps, for possible future contact with Alien cultures? 


Section Five - Special Purpose Companions 
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A Companion for learning in everyday life - Rebecca Eynon and Chris 
Davies 

Abstract 

The use of the Companion could have broad learning benefits. For example, enabling learners 
to have control over their projects of learning could potentially lead to improved level of self 
efficacy beliefs (Kim and Balyor, 2006), improved IT skills or a general confidence to try 
new things. We believe that this broader view of the benefits of such a Companion makes the 
positive implications outweigh the negative. 


There are times when any of us might prefer to find our own private way into making sense 
of some new knowledge, or perhaps to test out an area of possible future interest on our own. 
Not all learning is necessarily achieved with the help of a teacher or fellow learners, and 
some people have no choice in the matter: their circumstances mean that they have to cope on 
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their own, whether they are learning for personal interest, for fun, to gain qualifications, to 
expand horizons, to support life decisions or to update skills. For the solitary learner, the 
Internet and other new technologies can be invaluable in helping them to identify, plan and 
achieve their own projects of learning, so that they can achieve things on their own far more 
effectively than was possible previously. People can access new knowledge and create new 
content with the help of the Internet, and also connect with others when they feel ready to do 
so. 


However, despite these growing opportunities, the reality is that many individuals engaging 
in their own projects of learning are likely to use the Internet for learning in limited ways, and 
may often try but give up quickly. This can be for a number of reasons, perhaps due to a lack 
of skills in using the Internet effectively, or a lack of skills for learning, or having low 
confidence and self esteem as learners (e.g. Maki et ah, 2001; Kirby et ah, 2002). For the 
people who fall within this broad category especially, we believe that a strong argument can 
be made for developing a digital Companion that will in be able to ameliorate some of the 
problems that a solitary learner may encounter when entering into, or trying to sustain, self¬ 
generated and self-directed efforts at learning. By digital Companion, we mean some kind of 
embodied conversational agent on the computer screen which provides users with responsive 
and enabling interactions whilst using the computer that will help them progress towards 
increasing their control over their own learning; by providing encouragement, appropriate 
questioning, and persistent knowledge of ongoing explorations and discoveries. 


As the computer and the Internet have become more domesticated (Dutton and Helpser, 
2007), increasing attention has been paid to the potential of new technologies to facilitate 
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these kinds of learning opportunity, hopefully enabling some more substantial progress with 
the idea of a learning Companion, which itself is far from new. For decades, researchers have 
been exploring ways of trying to understand how best to use technologies to facilitate 
learning; and one avenue of research has been to explore initiatives where a computer can 
“teach” a student in a similar way to a real life educator. In relatively recent work in this area, 
researchers have begun to include animated pedagogical agents within intelligent tutoring 
systems (Chou et ah, 2003) to enable the learner to benefit both from personalised instruction 
and collaborative interaction (Goodman et ah, 1998). Pedagogical agents can differ quite 
considerably in terms of their interface, animation, level of intelligence, interactivity and role. 
One group of agents are called learning Companions, and are designed to have a more 
supportive role (the domain expertise is provided by another aspect of the system); and could 
act, for example, as a peer tutor, a teachable student, a collaborator, competitor or critic 
(Chou et al., 2003). 


In other ways the Companion proposed here is quite different from previous work in this 
area. First, the vast majority of the research has focused on the development of learning 
systems for school and college students, and has been based on specific aspects of the fonnal 
curriculum. Here, we are proposing a focus on adults learning for informal learning purposes; 
which can be defined as being “unorganised, unsystematic and even unintentional at times” 
(Coombs & Ahmed 1974), undertaken without externally imposed criteria or institutionally 
authorised instructors (Livingstone 2000), typically not leading to certification; crucially, 
learning which results from “daily life activities”, whether in work, family or leisure 
(European Commission 2001:33). Secondly, such systems tend to be developed around areas 
of the curriculum or specific domains, thus enabling parts of the system to teach the learner 
specific content. In contrast, in our vision of the learning Companion, the domain expertise 
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(traditionally provided by the agent and intelligent tutoring system) is provided by the use of 
the Internet and the topic is chosen by the learner. 


The type of Companion proposed here would act as a supportive, encouraging and responsive 
conversational partner that builds persistent knowledge of the user over time, referring back 
to the user’s previous interests, plans and comments. Yet, apart from teaching selected IT 
skills, the Companion will not be programmed to teach any domain-specific content 
knowledge. The subject expertise is provided via the Internet, thus the learning Companion 
would teach aspects of how to learn, how to interact with the material and people online, and 
how to produce new artefacts. This could involve the Companion helping the learner to sort 
through and tag new material for storage, visualise and graphically organise ideas, or 
encourage learners to review and articulate existing knowledge that might connect with the 
new knowledge, and to work towards a range of possible outcomes to their activities and 
projects, such as communicating findings on a topic of interest or importance to friends or 
family, polishing findings for inclusion in a personal portfolio or job/course application, 
publishing findings on the internet or archiving them for personal use. 


Like many pedagogical agents, such work falls within a social constructivist view of learning. 
That is, learning is seen as an individual and social activity - artefacts and other people form 
an important part of the learning process (Greeno at ah, 1996). Within this theoretical 
approach the use of technology for education is not just seen as a way to “deliver” content. 
Instead, the Companion would facilitate the interplay between the user, the computer and the 
Internet, and the culture and context in which learning takes place (Pea, 1993). 
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The approach proposed here also supports two important aspects of adult learning. First, the 
Companion places the onus on the adult learner to formulate their own goals of learning, 
make links between their existing knowledge and the new infonnation found, direct their own 
learning, ask their own questions and decide for themselves how well they comprehend what 
they have found out (c.f. Scardamalia and Bereiter, 1991). Thus, the whole process of 
learning is very much self-directed. Secondly as noted above, the learning Companion’s role 
will be to raise questions and create connections for users that help them to reflect on both 
“the content learned and the learning process” (Savery and Duffy, 1995: 33-34) and support a 
“reflective conversation with the materials of a situation”, and reflection on past reflection-in- 
action as a process which “may indirectly shape our future action” (Schon, 1983: 31). 
Through this conversation with the Companion learners can develop their met cognitive skills 
to support their learning. Indeed, previous work has used some form of learning Companion 
to encourage students’ articulation and self explanation (Aleven and Koedinger, 2002; 
Goodman et ah, 1998), reflection (Goodman et al., 1998), and emotional state (Burleson and 
Picard, 2004). 


The research that we are engaged in at the University of Oxford, where we are currently 
conducting Wizard of Oz experiments, aims to ascertain the feasibility of this kind of 
Companion for learning in everyday life 73 . The remainder of this discussion is divided into 
two parts. The first explores some of the design challenges of the learning Companion that 
have arisen during the research and the second considers some of the potentially negative 
aspects of a learning Companion that need to be explored. 
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Design challenges of a Companion for learning in everyday life 


When exploring the optimal design of any kind of learning Companion it is important to 
consider a range of learner characteristics (e.g. learner competency and age), potentially 
valuable theoretical perspectives and the findings from empirical studies. Work within this 
area is particularly challenging as much of the research tends to be relatively small scale, case 
specific and is difficult to generalise. Below we discuss some inter-related perspectives on 
design issues that are pertinent to our design of a learning Companion - the role of the 
Companion, the perceived competence of the Companion and aspects of the interface - 
before describing the kinds of functionality we would want to build into this tool. 


The role of the Companion 


The precise role the Companion takes within the learning process is a complex issue. Here, 
the role of the learning Companion is viewed as somewhere between a peer and an assistant, 
providing learners with a feeling of control over their learning by being responsive and 
reacting to their needs, but (particularly at the beginning of the learning relationship) offering 
quite a lot of guidance and support when required. The relationship between the user and the 
Companion is a difficult balancing act. As Lowe notes, artificial Companions “will need to be 
distinct and independent enough for us to treat their advice, information, and representation 
of our rules as real constraints on our action, but sufficiently closely aligned and sensitive to 
our goals that we do not feel them as an imposition” (Lowe, this volume). 
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The difficulty of this balancing act is apparent in terms of deciding how far the Companion 
should go in terms of acting as a peer rather than an assistant. The use of a peer metaphor has 
been felt to be the most appropriate to support adult informal learning and has also often been 
used in the development of Companions for formal education (e.g. Kim and Baylor, 2006), 
but at the same time we suspect that it should always emphasise some degree of its own 
subordination to the user’s needs. In earlier studies the research tended to focus on the agent 
learning alongside the learner, more recently the focus has been more about the development 
of a relationship between the learner and the agent. Some argue that the learner may help the 
agent as well as the agent encouraging the learner (Kim and Baylor, 2006; Kim, 2007). 
Indeed, in our conception of a Companion the relationship between the user and the 
Companion that builds over time is a key part of the process. Nevertheless, the precise role 
of a learning Companion for adults remains a significant challenge, particularly as we have a 
very limited understanding about what it means for people to leam via conversational agents 
(Veletsianos and Miller, 2008). 


The competence of the Companion 


The perceived competence of a learning Companion depends on a number of aspects, 
including the interface and the nature of the conversation. The focus of this section is on the 
conversation. As suggested above, the Companion will need to have conversations with the 
user about the process of learning and the topic of learning without having a great deal of 
knowledge about the latter. Thus a central issue for this research is whether the level of 
conversation can be convincing enough despite the lack of domain knowledge provided by 
the Companion. 
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Interestingly, other initiatives where the system did not understand the user have worked to 
some extent. For example, Weizenbaum’s (1966) ELIZA was relatively simple and did not 
have a great deal of depth, yet often gave an illusion of comprehending what the user was 
saying (Graesser et ah, 2004). More recently, a similar philosophy informed the design of 
AutoTutor. As the authors note, “the central assumption, in a nutshell, is that dialogue can be 
useful when it advances the dialogue and learning agenda, even when the tutor does not fully 
understand the student” (Graesser et ah, 2004:181). In our study the Companion may get to 
“know” something about the learner over time but it will never be a domain expert as in the 
case of AutoTutor. Yet, perhaps this expertise is not necessary to be perceived as 
knowledgeable about a topic. Indeed, humans seem to be willing to believe in computer- 
based agents; studies have shown that school and college students do perceive learning 
Companions as more intelligent than they actually are (Kim, 2005). Furthermore, although 
the results should be interpreted with caution, it appears that less academically strong 
students prefer and benefit most from a “low competency” learning Companion (Kim, 2007). 
Thus the target group for a Companion for learning in everyday life may be well suited to the 
type of agent that we propose here. 


In sum, a Companion that supports and guides learners in the ways described, without the 
need for a domain expert may well be possible. However, it is a big risk. If users lose faith in 
the competence of the Companion then this would be a significant problem causing 
frustration and perhaps total rejection of the system (Hietala and Niemirepo, 1998). 
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The nature of the interface 


Building a relationship between the learning Companion and the user is an important part of 
facilitating the learning process, in the same way as a student and a teacher or student and 
student need to build rapport in order to learn effectively from each other. Two important 
aspects of relationship building is the type of animation (if indeed there should be one at all) 
and the kinds of emotions the Companion displays and understands. 


A great deal of research has explored if a human-like animation supports the relationship 
between a user and the agent. Supporters of the use of human like agents argue that it is 
easier for humans to interact with human like characters as it reduces the efforts required in 
understanding and interpreting the communication. However, others argue that the use of 
human like agents leads to humans having a high expectation of the avatar which may not 
always be met and thus lead to problems (Dehn and Vanmulken, 2000). 


Indeed, in the context of learning there is a similar debate. In general, it has often been 
argued that the agent should be human like to support the learning process (Kim, 2007). A 
human avatar is expected by users to be more intelligent which may be beneficial or could 
cause problems for the type of Companion described here. In contrast, other research has 
shown that it is the dialogue that has an effect on learning not the presentation of the avatar 
(e.g. Graesser et ah, 2003) and that when students were asked about their perceived 
preferences the quality of the interaction was more important than the design or aesthetics of 
the virtual character (Mahmood and Ferneley, 2006). 
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However, this area of research suffers from methodological problems as a great deal of the 
work has been conducted on college students in short-term studies. Thus, the preferences in 
adults over a longer period of time are difficult to ascertain. As Dehn and Vanmulken (2000) 
note, the preference or value in having a human like avatar depends very much on the goals 
of the system and the user. Instead of asking if the use of an animated character has a positive 
or negative effect on learning the question should be “what kinds of characters have 
positive/negative effects for which.groups of learners” (Gulz, 2005: 417). 


A second question centres on emotions. Emotions are an important aspect of learning, thus 
need to be considered within the design of a learning Companion (Kort et ah, 2001). When 
we are trying to learn something new we may go through a variety of emotional states 
including frustration, anxiety, hope and enthusiasm. Negative emotions such as frustration 
can often be part of the learning process yet if not supported appropriately the learner may 
just see themselves in a negative light (e.g. stupid or incapable) and thus give up or not leam 
anything. Kort and colleagues argue that a good teacher or mentor can recognise these 
emotional states in the people they are trying to teach and support them. They argue that 
computer mentors also need to read the affective state of the user and respond appropriately if 
they are to effectively support the learning process (Kort et ah, 2001) and ensure users 
persevere with a particular project of learning (Burleson and Picard, 2004). 


While not a specific focus of the Companion proposed here, affective agents (i.e. agents that 
understand users’ affect and respond showing affect) could be used to encourage users to 
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develop metacognitive skills about their affective self-awareness and find ways to deal with 
negative emotions that emerge from trying to tackle difficult tasks (Burleson and Picard, 
2004). In other words, “a learning Companion that will not be an ITS agent that already 
knows the task domain, but rather, acts an affective partner, on the side of the learner - a 
collaborator to assist in affective self awareness. One approach we believe will be particularly 
fruitful is to present the affective agents as one or more peer-learning Companions” 

(Burleson and Picard, 2004:3). Such an approach may be particularly important within some 
groups of adult learners who may well encounter many negative attitudes towards learning 
when trying to explore something new. 


All of the perspectives considered above indicate that research still has a great number of 
issues to deal with in respect of exploring the characteristics or combination of characteristics 
of a Companion that may best facilitate learning (Graesser et ah, 2004), especially with 
respect to different groups of learners (Kim and Baylor, 2006). Given this complexity the 
design of such a Companion needs to be flexible and adaptable to different learners or 
alternatively incorporate a number of agents (e.g. an expert and a mentor) within the learning 
environment that the learner can select from (Kim and Baylor, 2006; Gulz, 2005; Hietala and 
Niemirepo, 1998). However, in our proposal we hope that one Companion can be sufficiently 
flexible to meet these different requirements, as the relationship between the Companion and 
the learner is of paramount importance. 


At the present stage, we can go no further than indicate, as we do below, what we consider to 
be the key aspects of behavior that might be feasible, that we would wish to build into the 
early iterations of our learning Companion. 
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The learning Companion in action 


We see the achievement of the five ongoing tasks for the learning Companion, outlined 
below, as necessary minimum requirements for a Companion to provide support solitary 
learners that does not involve pre-programmed content knowledge. These would involve the 
Companion operating primarily through a process of questioning that enables it to articulate 
and check back on its understanding of the user’s interests and goals. Any guidance and 
encouragement that it provides will depend entirely on its capacity to elicit and itself use the 
language of the user’s expressed agenda: 


1. The learning Companion must enable users to identify, articulate and categorise areas 
of interest: this is likely to involve a number of repeated stages in which the 
Companion encourages users to name any past, current or potential topics or areas of 
knowledge that they might wish to explore further. This will involve a process of 
focusing down through processes such as tagging and categorising, and checking back 
with the user, in order to establish stable and useable terms. The process would aim for 
more than one area of interest, so that the Companion has something to refer back to at 
a later date. 

2. It must lead users towards adopting an explicit orientation towards learning: the 
learning Companion will work progressively towards using the word learning in 
dialogue with the user, increasingly emphasising that the enterprise will have longer 
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term value than simple infonnation seeking; through its questioning and feedback, it 
should help to build a large picture of the learning to be undertaken, so that users see 
their activities as more than transitory. 

3. It must help the learner to construct a plan of action: as the project grows, the 
Companion will help the user to identify specific end-goals: a personal record of what 
has been learnt; some means of communicating what has been learnt to others; or the 
willingness to join a relevant community of interest, either online or offline. The 
Companion must therefore be capable of suggesting a limited number of options, and 
asking questions that will help users themselves to plan a sequence of actions and 
mini-goals through completing a wiki-type template initiated by the Companion; 

4. It must keep the learner on track: various approaches must be tried out in order to 
remind and encourage learners to persist with selected goals and plans. Each session 
might end with a clearly articulated mini-goal, for instance, with which to begin the 
subsequent session. The persistent nature of the learning Companion will enable it to 
draw on the plan of action in order to remind the user of what was agreed, and to ask 
how that is going, and collect information about any progress achieved, and providing 
affective support and approval. 

5. It must be responsive to the learner’s level of engagement in the task: the Companion 
will need to ask questions which allow it to recognise when a particular project should 
be adapted, abandoned or acknowledged as complete. At that time, the Companion 
might need to refer back to topics previously mentioned but not yet taken up, or ask 
about new topics for potential interest that have subsequently come to mind, or help 
the user to connect with wider contexts of learning. If the user is still working with the 
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Companion, then the most likely move would involve starting on a new cycle of 
learning. 


Potential negative aspects of learning Companions 


Developing a Companion for learning in everyday life leads to broader questions about the 
individual, technology and society. Here two issues are explored: 1) the extent to which such 
a Companion may encourage solitary learning and 2) what happens if the Companion 
facilitates incorrect learning. 


Some people express concern that artificial Companions may become a substitute for real 
human interaction. As Cowie (this volume) asks, “what would happen if a Companion was 
too engaging, thus people become more socially isolated as they prefer their Companion to 
people?” A similar question can be posed about education and learning, what if people 
increasingly chose to learn by themselves? How much of a problem could this be? 


Current thinking about learning emphasises the essentially social nature of successful 
learning, and the importance of building understanding through dialogue and collaboration 
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with both teachers and fellow learners. Within such a perspective, the most powerful digital 
tools for learning are seen as being those that support, stimulate and sustain collaborative 
networks for learning. We would not argue with this but, as we indicated at the start of this 
chapter, there are probably many people who, through personal circumstance or inclination, 
would prefer to benefit from learning encounters of a non-human kind as part of a long 
process of developing confidence and competence as learners. Interaction with the 
Companion may lead to human interaction online and in the longer term using the 
Companion might lead them eventually towards organised learning environments. The first 
step (and, for some, the only step they might choose to take) might most productively be to 
engage with a Companion in order to leam things on their own, and on their own terms. 


Like other writers in this area, we are suggesting that the social aspect of learning might, in 
some key respects, be recreated for an adult learner through the use of a digital tool. For 
example, while Kim and Baylor are careful to point out that simulated peer interaction is not 
equivalent to “real life” social interaction with peers or a replacement for it, they argue that 
such interaction may be good enough to facilitate learning (Kim and Baylor, 2006). The use 
of a digital tool to enhance or replace some aspect of social learning is not a particularly 
radical thought in itself, but it does raise the rather more radical implication that, for some 
people, this might prove to be what they want, and all that they want. In effect, we suspect 
that the digital learning Companion could demonstrate that solitary learning is an acceptable 
concept. The longer tenn implications (both positive and negative) are difficult to ascertain. 


Secondly, there is the issue of a learning Companion possibly facilitating incorrect learning, 
which is not something that can be overcome if the focus is on supporting self-directed 
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learning on any topic available on the Internet. Unlike other pedagogical agents, our 
Companion would not be able to assess if learning was correct or not. This is in direct 
contrast to other work in this field. For example, AutoTutor asks questions on pre-specified 
topics and then from the qualitative answers given provides a range of responses that can 
facilitate learning. These responses include: feedback (positive, neutral or negative), prompts, 
hints, and the identification and correction of mistakes (Graesser et al., 2004). The 
Companion proposed here would not be able to make these judgements. For some, this may 
be unacceptable. However, in reality people are already learning (both correctly and 
incorrectly) in their everyday lives both for formal and informal purposes. While the content 
of learning may not always be correct, it is likely that it will be at least some of the time. 


Conclusion 


Ongoing work with Embodied Conversational Agents indicates that human users are easily 
engaged in relationships with machines for a variety of purposes, and it is reasonable to 
suppose that such relationships might prove to be highly productive for certain types of 
learner, in certain types of settings. As yet, there is very little research evidence to enable us 
to understand how users will in fact communicate with these conversational agents for 
educative purposes, or how this can be made to work in an engaging, sustainable and 
productive way for them. Being content free, the tool’s main functions will be to help users to 
give voice to, and make plans for, learning projects that might otherwise have remained 
unrealised without its Companionship and support. At this minimal level, the main functions 
of the learning Companion will be its capacity to stimulate a potential learner’s thinking 
about possible learning projects, and then to help them move forward through projects by a 
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combination of machine-like persistence and human-like qualities of friendliness and 
encouragement. 


The Maryland virtual patient as a task-oriented conversational Companion 

- Sergei Nirenburg 


Abstract 

This chapter describes a conversational agent environment, the Maryland Virtual 
Patient (MVP). MVP models the process of disease progression, diagnosis and 
treatment in virtual patients endowed with a “body,” a simulation of their 
physiological and pathological processes, and a “mind,” a set of capabilities of 
perception, reasoning and action, that allows the virtual patient to exhibit independent 
behavior, participate in a natural language dialog, remember events, hold beliefs about 
other agents and about specific object and event instances, make decisions and learn. 


Introduction 

Building digital Companions is clearly a subfield of the flourishing field of research on 
intelligent agents. The purview of this field is rather broad and covers such diverse concerns 
and approaches as: Partially observable Markov decision processes; modelling agent teams 
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and agent negotiation strategies; agent-agent and agent-human communication; and decision- 
theoretic issues and algorithms related to planning and rational action. Companion-oriented 
work shares many of the above concerns, though it is less focused at the moment on team- 
level issues. However, whereas in general even some rather simple programs can be labelled 
intelligent agents, Companions are typically understood to be complex and multifaceted 
systems that emulate human levels of behavior. It is broadly believed that the utility of a 
Companion application should be proportional to its ability to keep the user happy (cf., e.g., 
Cowie, this volume). It follows that making mistakes in comprehension and reasoning is not 
really a problem for a Companion if it succeeds in being attentive to desires and preferences 
of the user. In short, it is less important how well a Companions is at a particular task as long 
as it succeeds in having the user suspend his or her disbelief and treat the Artificial 
Companion as if it was a living human Companion. The above does not hold for task-oriented 
intelligent agents that must succeed in performing the tasks for which they are designed in 
addition to upholding the verisimilitude in communicating with the user. While a Companion 
that approximates a human might be the ultimate goal of Companion development, it is, thus, 
understandable that for practical reasons Companion systems stress the development of 
functionalities that maximize the chances for the suspension of disbelief and promote the 
perception of the verisimilitude of user-system interaction. 

In this section we first consider the various goals that can be pursued in building Companions 
and what it would take to make users satisfied with Companions, then discuss some 
methodological choices relevant to building computational Companions. 


Achieving user satisfaction. 
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It can be argued that a Companion’s concern with the user’s physical well-being is embodied 
in the first two of Isaac Asimov’s three laws of robotics. 1 Indeed there has been a strain of 
agent research that has been inspired by this famous science fiction reference - cf. Weld and 
Etzioni 1994, Gordon 2000, Pynadath and Tambe 2001, Schurr et al. 2006. 2 Note, however, 
that Asimov’s precepts - unlike the U.S. constitution - do not touch on the pursuit of 
happiness. To discuss how a Companion can make a user happy, one should first try to 
understand, at least in simplified terms, what “a user being happy” involves. It is for this 
reason, I think, that it is important, as contributions in this volume do, to concentrate not so 
much on the capabilities of current and future systems but rather on what properties of 
computer systems will be necessary if people are to accept them as true or at least acceptable 
Companions. There is, of course, anecdotal evidence that people would endow even the 
silliest technological concoctions with consciousness and/or intentionality - cf. the 
Tamagotchi phenomenon, the reactions to ELIZA (by Weizenbaum’s secretary, who guarded 
her secrets from ELIZA, or by a protagonist in David Lodge’s Small World) or the beliefs of 
Turkle’s (1985) subject Anthony (see Levy, this volume). To attract more sceptical 
technology adopters, formal user studies must be conducted (argued for, among others, by 
Newell, this volume). Wizard of Oz experiments are among the most popular methods of 
simulating system performance in the area of dialogue. 


But what will it take to construct a useful Wizard of Oz experiment to gauge user satisfaction 
with a Companion? The person playing the role of the wizard must be able to “simulate” the 
various core properties that the Companion must have. So, these properties must be analyzed. 
A few such properties have been suggested and discussed by the various contributors to this 
volume.’ Thus, Wilks lists some properties desirable in a “Victorian” Companion - 
politeness, discretion, modesty, wit, cheerfulness, trustworthiness, etc. Wilks suggests this list 
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at least in part to juxtapose it against the preferences inspired by the field of affective 
computing, where emphasis is, understandably, on Companions being able to demonstrate 
emotions. Wilks’ feature list can be expanded, for example, by adding character traits (e.g., 
shyness or obstinacy or earnestness) so that the user has a choice of Companion personality. 4 
Also, Wilks understandably omits any overt mention of features that are present in people but 
cannot be simply assumed in Companions - basic capabilities of perception, reasoning and 
(to a lesser degree) action, including the important capabilities of learning by being told and 
collaborating on tasks. Some of these basic human features are listed by Pulman (this 
volume; we will return to them in the conclusions of this chapter). It is also clear that in 
discussing these features one must pay attention to potential conflicts - politeness may 
interfere with trustworthiness, etc. 


From human-oriented descriptions to implementable features. 


A more important thing to notice about the items on the Wilks list is that before they can be 
used in computational modelling they must be first given an interpretation. This is because 
most of these items are essentially metaphorical or refer to metaphorical concepts (this is a 
different, less general, reference to metaphoricity than the one meant by Bryson, this 
volume). This situation brings to mind Minsky’s (2006) “suitcase words” that have a vague 
meaning but refer to important though not universally clearly defined (or definable) concepts, 
such as consciousness, emotion, perception, awareness, intelligence, attention or, for that 
matter, happiness. Minsky points out that it might be counterproductive to try to make the 
corresponding notions more precise: “we need to use those suitcase words in our everyday 
lives to keep from being distracted by thinking about how our thinking works” (2006: 128). 
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This approach licenses our everyday use of metaphorical definitions - as when we say that 
somebody is boiling with indignation or has the eye of an eagle when it comes to 
impropriety. It is intended for people and is expected to be broadly understood without 
precise definitions. 5 The implication is, it must be presumed, that for building actual systems 
one will have to make the definitions of the objects of modelling more precise. 


How many times have we read scholarly articles declaring that a problem has been solved or 
a phenomenon has been covered only to conclude that the relation between the phenomenon 
in its entirety and what is actually implemented is metaphorical, that in reality but a small 
fraction or facet of the problem has actually been treated? That this can be perceived as false 
advertising may be blamed on (sometimes, no doubt, intentional) ambiguity between the 
metaphorical “suitcase-word” and a more sober constraining definition of tenns. For 
example, the capabilities of agents (“virtual humans”) in the SASO-ST system, as described 
in Traum et al. 2005, include modelling trust. Trust is, in turn, operationally modelled using 
the categories of familiarity, solidarity and credibility. In SASO-ST solidarity is calculated on 
the basis of the percentage of the user’s assertions or demands that are congruent with the 
agent’s goals (how does one calculate congruency?). Credibility is interpreted as a count of 
how many times the user makes assertions consistent with the agent’s beliefs. Familiarity is 
understood as a measure of obeying the nonns of politeness (alas, the authors do not dwell on 
how these are interpreted). While this modelling of psychological categories is quite possibly 
the most interesting conceptual part of the system, it is equally clear that trust-related 
computations in the system differ from the everyday “suitcase-word” interpretations. 
Credibility and solidarity are suitcase categories: using them to describe the actual properties 
computed is metaphorical. And familiarity is used to describe another suitcase category, 
politeness. 
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Sets of Companion features and functionalities that succeed in making the users suspend 
disbelief can be fully determined only empirically. As to R&D of systems that could be 
subjected to such experimentation, the strategic choice must be made somewhere between the 
extremes of a) integrating currently available mature technologies that offer guaranteed but 
limited-quality results and b) postponing the integration of Companion systems pending 
improvement of currently immature or under-evaluated technologies that show promise of 
fulfilling the prerequisites for successful Companions (by offering arguably better chances 
for user acceptance). Prominent among the immature technologies are near-human-level 
language processing and reasoning. 


The hurdle of natural language processing. 


Natural language processing is a notoriously difficult problem that has survived a number of 
paradigm shifts and concomitant surges of optimism in the community and the general 
public. A lot of progress has been made in NLP over the past forty years or so; so it is 
somewhat puzzling to find that the basic underlying technology in question-answering and 
even practical dialog systems at the moment has much more in common with ELIZA than 
one would expect: connecting questions with answers is still done using essentially matches 
on textual strings, not representations of the text meaning (see below an illustration of NLP in 
state-of-the-art virtual patient systems). Of course, these days the matches may be 
contextually infonned and conditioned on empirically established significant cooccurrences. 
But the core principle is still the same. So, Companions researchers may either seek to bypass 
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this problem by anchoring the language capacity of Companions in empirically derived 
probabilistic models or concentrate on core NLP problems having to do with meaning 
extraction and manipulation. 6 The choice will depend on the judgment of which strategy will 
bring user acceptance faster and at a smaller expense, as well as on the purely scientific 
preferences of specific researchers who might be interested in the language understanding 
problem irrespective of an application. A number of hybrid strategies in which advanced 
functionalities are used partially are also possible. For example, advanced language 
processing can be implemented at first for a specific application domain and then extended to 
other domains. The viability of this methodology has been a long-standing hypothesis of 
knowledge-based language processing that is still awaiting a serious test. Recent advances in 
knowledge acquisition methodologies and machine learning make the feasibility of such a 
test more realistic than ever before. 


Realism in expectations. 


Whatever the position of the individual researchers in the field, all will agree, I think, that it 
is essential not to oversell the current and near-future capabilities of computational 
Companions. Sloman (this volume) wisely cautions against overselling Companions by 
noting that their current inability to deal with novel situations may lead to accidental 
catastrophic results, which may ruin the field before it flourishes. To guard against this 
danger, one may suggest such safeguards as limiting the autonomy of a Companion and 
making Companions capable of learning by being told. 74 Moreover, limiting the capabilities 
of Companions and engineering task-oriented Companions (Wilks, this volume, mentions 
Senior Companion and Health and Fitness Companion demonstrators) may serve both as a 
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safeguard against undue hype and as a promising business model for the times when 
computational Companions will be mass-produced. On the downside, the purpose of task- 
oriented Companions may be broader than just keeping the user happy and prepared to 
suspend disbelief- the Companion must also facilitate the performance of its chosen task. 


A computationally feasible approach to building advanced Companions. 


This chapter presents a case study of an implemented multi-agent environment that seems to 
meet the desiderata suggested by Pulman (this volume) for conversational Companions. The 
environment currently serves as a Companion in a specific application domain (clinical 
medicine) and for a specific task (providing the human user with computer models of patients 
for training purposes). However, its architecture, knowledge resources, and processors are 
domain- and task-independent and can readily be adapted to any domain and many tasks, 
including those more useful for general-purpose conversational Companions. The core 
intelligent agent in this environment is a virtual patient (VP) that can communicate with the 
human user (typically, a physician in training) in English, make decisions about its health 
care, ask for information, learn information, create and use memories, and cognitively and 
physiologically function over time in a simulation that is sufficiently robust to be used for 
training purposes. This environment has been under continuous development since 2006 in 
the Maryland Virtual Patient (MVP) project. The MVP environment represents a departure 
from state-of-the-art medical training systems and, we believe, represents a viable model for 
developing more advanced conversational Companions. 
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MVP goals and priorities. 


The immediate impetus for the theoretical, descriptive and system-building work in the MVP 
project has been the need to improve the process of educating future physicians. More 
specifically, the project concentrates on creating technological means of alleviating three of 
the limitations inherent in current medical school curricula: 


1. Future physicians do not have the opportunity to adequately practice their cognitive 
analysis and problem solving skills, especially with regard to treating non-emergency 
patients over long periods of time. . 

2. During their apprenticeships, future physicians typically do not see a) patients with all 
the diseases that must be studied or b) a sufficient number of diverse cases of a 
particular disease. 

3. There is not enough time for teachers to spend with individual students and it is 
economically infeasible to hire more teachers. 


In contrast to all current and previous VP-oriented systems (cf. review in Section 3), the work 
in the MVP project is devoted to immersing human users in simulated environments. 
Therefore, we concentrate on the breadth of coverage and realism of the simulation and 
postpone the discussion of such architectural and control issues as resource allocation, 
efficiency of agent collaboration and task scheduling. This deferral is made possible by 
making a variety of simplifying assumptions in our system. For example, we assume that the 
scheduler, executor and the demons have all the time they need to complete all their 
operations at a given agenda time slot. In other words, virtual time is not equated with real 
time. Our simulations are not at this time negatively affected by this assumption because we 
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have been dealing with chronic diseases that unfold over long periods of time. We will revisit 
this simplifying assumption when we start modelling acute diseases where durations of 
crucial events can be very short. Similarly, our application environment allows for full 
availability of not only computational resources but also of human resources - such as 
additional members of the medical team: specialists, lab technicians, nurses, etc. - in the 
world being modelled. In the current version of the system these additional agents are 
modelled in a coarse-grain fashion only. The system’s quality and utility does not suffer if 
we do not concentrate on scheduling particular specialists or lab technicians for particular 
tasks, even though the kinds of interactions that the user has with the various intelligent 
agents are currently not fully covered. In the future we intend to remove these simplifying 
assumptions . 


Toward the next generation of virtual patients 


To make VPs more sophisticated, The MVP project has pursued four interleaved directions of 
work: a) simulating normal and pathological physiological processes; b) modelling human 
cognitive, including conversational, capabilities; c) formalizing the clinical experience of 
master physicians and using the knowledge of these “best clinical practices” to support 
automatic tutoring; and d) integrating the VP in a computational environment modelling a 
society of intelligent agents. 


The desire to enhance verisimilitude through increased modelling sophistication carries a 
steep price tag. As an illustration, compare the amount of work carried out by the MVP 
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system and a representative state-of-the-art VP system (Chesher 2004) to respond to a dialog 
turn of the user. The latter system (and most of the other state-of-the-art VP systems as well 
as dialog-capable training systems in other fields, cf. Schurr et al. 2005) relies on matching 
keywords from the input with a list of inputs for which it has prepared answers (dialog turns 
to be generated by the system). By contrast, in order to generate a comparable dialog turn, the 
VP in the MVP environment has to perform a large number of non-trivial operations. For 
example, to answer the “simple” query How are you feeling?, it must: 


1. Extract the meaning of the user’s dialog turn, including its illocutionary 
force, i.e., the speech act meaning 

2. Add the resulting text meaning representation (TMR) to the short-term memory 
component of its fact repository 

3. Generate an instance of a “Be-a-Cooperative-Conversationalist” goal 

4. Prioritize goal instances on its active goal agenda 

5. Select a goal instance for processing 

6. [Once the above goal instance is chosen] select a plan to pursue to attain this goal; there 
is currently just one plan for this type of goal: “carry out a relevant verbal action” 

7. Specify content of the verbal action to be produced. The VP must recognize that it is 
playing the role of patient in an instance of the “MD visit” script; look up in its fact 
repository either a) the value of the property “health-attribute” or b) the value of the 
gravest symptom recorded in its fact repository. Which symptom is the gravest is a 
function of kind of the kind of symptom and its severity 

8. Generate an English sentence that realizes the above content, which is a report about 
its health attribute. In the current version, lexical selection is based on the value of health- 
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attribute and syntactic structure selection is analogy-based, driven by random selection 
from an inventory of sentence patterns. 


We believe that developing the capabilities to support the above activities is necessary for 
high-quality digital conversationalists, even at high cost. Once we have high-quality 
conversationalists, porting them to different domains is relatively simple since the most 
difficult aspects of language processing are found in any domain and practically any genre. 
We also believe that attaining this level of sophistication on a realistic scale is feasible. We 
believe that the statements often heard nowadays about the demise of AI are ill-conceived. 
The AI enterprise did not fail. In fact, it has not yet been brought to the test. This is because 
the enterprise is much more complex than it was perceived to be even by many AI 
practitioners themselves. Despite the recent emphasis in the field on statistics-oriented 
methods, they should not be viewed as having superceded classical AI. In fact, these methods 
have contributed to the core task of knowledge acquisition that is a prerequisite to the success 
of the program of AI. Progress in learning, knowledge visualization and other ergonomic 
factors, the ease of access to vast collections of data on the Web and other developments 
make the original AI goals incrementally more attainable. Our work on the MVP corroborates 
this state of affairs. We believe that the development of a comprehensive MVP is feasible 
both scientifically and logistically. 


The top-level structure of the MVP environment is illustrated in Figure 1. The human agent, 
who is typically a medical professional or medical student seeking to improve his or her 
cognitive decision making skills, plays the role of the attending physician. Artificial agents 
include the VP, other medical personnel - such as lab technicians and specialist consultants - 
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and an automatic tutor. In this description, we will concentrate on the two sides of the VP 
agent - the physiological agent and the cognitive agent. 


The VP itself is a knowledge-based model and simulation of a person suffering from one or 
more diseases. The VP is a “double agent” in that it models and simulates both the 
physiological and the cognitive functionality of a human, something necessary for the domain 
of clinical medicine. Physiologically, it undergoes normal and pathological processes and 
responds realistically both to expected and to unexpected (e.g., user error-caused) internal 
and external stimuli. Cognitively, it experiences symptoms, has lifestyle preferences partially 
based on a model of character traits, has dynamic memory and learning capabilities, has the 
ability to reason in a context-sensitive way, and can communicate with the human user about 
its personal history, symptoms and preferences for treatment. User-VP communication is 
carried out in English. 


Virtual Patient 


Other Medical 
Personnel 
Agents 


Physiological 
Agent 




< 


Cognitive 

Agent 


Tutor 

Agent 


User Interface for Attending Physician (Trainee) 





Figure 1. The network of agents in MVP. 
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In the MVP project the emphasis has been on making the VP more realistic by modelling it as 
an intelligent agent that is capable of human-like behavior in situations that are not rigidly 
pre-scripted. The modelling approach used in MVP is transparent and explanatory in the 
sense that simulation of human behavior is based on formal encoding of theories of its many 
component processes and not purely on empirical stochastic induction on the basis of a 
corpus of human dialog turns. 

Another direction of making the medical student’s learning experience more realistic is to 
track changes in the patient’s (patho) physiology over time, with special attention paid to 
changes that may be induced by the simulated interventions prescribed by the student. Note 
that the interventions prescribed (both inappropriate and appropriate ones) may trigger the 
appearance of side effects that effectively amount to new diseases that need treatment in their 
own right (see, e.g., McShane et al. 2007). It is practically impossible to anticipate all the 
possible treatment sequences and their side effects. As a result, authoring scenarios (such as 
those used in first-generation VP systems) for the entire space of eventualities even for a 
single disease is impractical. 


The physiological agent 


The subject domain of the MVP project is clinical medicine. The core tasks in building the 
physiological agent are a) developing clinical theories of disease progression and b) 
implementing the theoretical parameters and their fluctuations in a knowledge-based 
simulation environment. We call our theories clinical to underscore the fact that many of the 
physical, chemical and physiological processes and their causal connections are either not 
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understood or are not necessary to support realistic simulation. Whenever such processes are 
encountered in describing disease progression, we use “bridges,” elements of clinical 
expertise that distil expert knowledge of medicine. The resulting model, thus, constitutes a 
medically correct approximation of a truly causal model of disease progression. 

The availability of a simulated physiological model allows us to: 

• forgo specific authoring of a VP’s state of health at a particular time, including effects 
and side effects of interventions and treatments 

• derive the results of a variety of laboratory tests directly from the parameters of the 
“body” of the VP, and 

• make the intelligent agent ever more “human-like” by facilitating the modelling of 
interception - the conscious perception of symptoms by the model of VP’s “mind.” 

The physiological side of the VP is modeled as a set of interconnected ontological objects 
representing human anatomy and ontological events representing human physiology , 
including pathology. Each object is described by a set of ontological properties and their 
associated value sets. Crucial among the properties are those that link the objects to typical 
events in which they participate. These events are usually complex - that is, they include 
other, possibly also complex, events as their components. Following Schank and Abelson 
(1977), we call these complex events scripts. Examples are swallowing by a healthy VP, 
swallowing by a VP with a diseased esophagus, the process and effects of interventions, 
clinically appropriate ways to manage patients with given signs and symptoms, etc. 

At any given time, the model of the normal human contains whatever nonnal anatomical and 
physiological knowledge was compiled to cover the diseases currently available in the 
system. So, although at present our virtual humans do not have a highly developed model of 
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the circulatory system, as soon as we have completed the circulatory model - which is 
currently under development to support the modeling of heart disease - all virtual humans 
will be endowed with all the associated functionalities and property values. 

In MVP, diseases are modeled as processes that cause changes in key property values of a 
patient over time. For each disease, a set number of conceptual stages is established and 
typical values or ranges of values for each property are associated with each stage. Relevant 
property values at the start or end of each stage are recorded explicitly, while values for times 
between stage boundaries are interpolated. The interpolation currently uses a linear function, 
though other functions could as easily be employed. 

A disease model includes a combination of fixed and variable features. For example, 
although the number of stages for a given disease is fixed, the duration of each stage is 
variable. Similarly, although the values for some physiological properties undergo fixed 
changes across patients, the values for other physiological properties are variable across 
patients, within a specified range. The combination of fixed and variable features represents, 
we believe, the golden mean for disease modeling. On the one hand, each disease model is 
sufficiently constrained so that patients suffering from the disease show appropriate 
physiological manifestations of it. On the other hand, each disease model is sufficiently 
flexible to permit individual patients to differ in clinically relevant ways, as selected by 
patient authors. 

At the time of physiological model development, knowledge engineers help physicians to 
distill their extensive and tightly coupled physiological and clinical knowledge into the most 
relevant subset, express it in concrete tenns, and hypothesize about unknown and currently 
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unknowable things, like the nature of disease progression at the pre-clinical (pre- 
symptomatic) stage. The models thus created are very different from anything physicians 
themselves have ever encountered and anything that is available in the medical literature. 


The cognitive agent 

The cognitive side of the VP is at present capable of two types of perception: interception and 
the perception of linguistic input from the human user. Responses to interceptive input are 
remembering a sensation (typically a symptom) and deciding whether or not to do anything 
about it at the given time. Responses to language input include learning (augmenting the 
agent’s ontology, whenever appropriate, as a result of understanding user input), responding 
to a question or suggestion, generating a question based on infonnation or advice just 
provided, and remembering the content of dialogs. 

MVP uses agenda-style control and goal- and plan-based simulation. At present, the 
physiological and cognitive agents of the VP share an agenda and the knowledge resources. 
This architectural decision may be changed in future versions. We represent goals as 
ontological instances of properties. MVP relies on large knowledge bases that include a 
language-independent ontology of over 10,000 finely described concepts, including the 
scripts described above; a lexicon, mapped to the ontology, of over 30,000 senses that are 
expanded at runtime to several times that using productive word-formation rules; a “memory” 
for each VP that can be populated on the fly during a simulation; and a suite of processors for 
language processing, reasoning, simulation, and so on. 
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A core architectural aspect of the system is that all interoception and all language input are 
automatically translated into expressions in the same metalanguage of memory and reasoning 
used by the VP. As a result, all cognitive processes are modeled using formal, unambiguous 
knowledge structures that are grounded in the ontological model of the world. Figure 2 
presents a somewhat simplified illustration of the architecture of the cognitive agent inside 
the VP. 



Figure 2. The architecture of the cognitive agent of the virtual patient in the MVP project 
(simplified). 


Each VP instance has its own ontology, lexicon and memory, reflecting its own cognitive 
reality, and each can learn over time through the simulation. For example, the VP can learn 
information about its disease, possible management strategies, etc., from the user, thus 
expanding - or changing, if there were previous misconceptions - its ontology and lexicon 
over time. The state of the VP’s knowledge affects all of its decision-making functions. 
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The VP contains a set of beliefs about each of the other agents, with the exception of the tutor 
of whose existence it is not aware, that is formulated in exactly the same way as the 
knowledge about the VP - though it might be much more sparsely populated. The VP’s 
“profiles” of other agents include their ontology’s, fact repositories and lexicons as well as its 
assessment of the character traits, physical and mental states of these agents as well as their 
perceived attitudes toward various entities in the world, centrally including the VP itself 
(“this secretary is rude to me”). Economy in recording this information is a separate issue: for 
example, the VP may assume that some other agent’s ontology and lexicon are almost 
identical to its own ontology and lexicon and just mark the few differences. 


The VP also has a set of beliefs about self that can be different from reality: for example, it 
can believe itself to be, say, more truthful or more healthy than it is in reality. Some of VP’s 
actions will, therefore, be guided by its real self and some, by its opinion of self - this 
capability is not yet implemented but is an interesting capability to model in a virtual human, 
for example, to highlight discrepancies between its behavior and its beliefs about self. The 
“real” profile of the VP is visible only to system developers but may be revealed to the 
human user if a teacher considers this pedagogically beneficial, 


VPs in MVP vary with respect to the extent of their knowledge of medicine, the content and 
accuracy of their memory of past events and their personality traits. This enables variation 
across VP instances in decision making and the content and form of dialog acts which, in 
turn, enhances the verisimilitude of training encounters. VPs can initiate actions, which also 
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enhances the verisimilitude of training encounters by forcing the user to react to a patient’s 
potentially unexpected behavior. VPs are capable of introspection in that they can explain 
why they are doing what they are doing. VPs are also capable of learning. As a result of their 
communication with the user, they can leam facts, ontological concepts with their properties 
and lexicon entries. These learning-inducing dialogs can be initiated either by the user or by 
the VP. In the latter case, the process originates with instantiating an instance of the goal 
“know-infonnation-about-concept” and following the plan that involves asking the user 
informational questions. In the future, learning by reading and learning by reasoning will be 
added to the repertoire of the VP. 


Perception, Reasoning and Actions of the Cognitive Agent 


75 


Interoception is the process through which the cognitive agent becomes aware of some of the 
states of the physiological agent. The result of this awareness is recognizing and 
remembering a symptom. The communication channel between the physiological and the 
cognitive agent is narrow: the cognitive agent is not fully aware of the activities of its “body,” 
the physiological agent. In the current version of MVP, the interoception module operates a 
set of demons that are programmed a) to notice the changes in values of specific 
physiological parameters and b) if these values move outside a certain range, to instantiate 
corresponding symptoms in the VP’s memory - more specifically, in its short-term memory 
component. 

Symptoms are represented as values of properties in the cognitive agent’s model of self. This 
model is realized as an instance of the ontological concept human and stored in the cognitive 
agent’s fact repository. In addition to the model of self, the VP also has models of other 
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agents of which the VP is aware, such as each of the MVP users. Note that the properties in 
this cognitive profile are different from the properties used in the physiological agent. For 
example, the latter will include the property “pressure in the lower esophageal sphincter” 
while the former may record a certain level of the property “heartburn,” without necessarily 
knowing that heartburn may be affected by a certain level of pressure in the lower esophageal 
sphincter. 

When certain symptoms appear and reach a certain level of severity, the health-attribute 
property in the cognitive agent’s profile of self decreases. This attribute represents a 
generalization over all symptoms. A decrease in the value of the health-attribute causes the 
appearance on the agent’s agenda of an instance of the goal “be-healthy.” 


Perception through understanding of natural language utterances is a major contributor to 
enhancing the verisimilitude of a VP encounter. As mentioned above, the language 
understanding module extracts semantic, pragmatic and discourse-oriented meanings from 
inputs and represents them in the same fonnalism that is used by the reasoning module. This 
functionality is useful a) because it facilitates reasoning on the basis of language input and b) 
because it supports reasoning that is necessary for understanding language input in the first 
place. 


Language understanding in the MVP system is based on the OntoSem text analyzer, which 
integrates a large collection of algorithms and knowledge resources. The list of algorithms 
used in the current version of the MVP just for basic semantic analysis includes: 
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• an ambiguity resolution algorithm based on multiple-value selectional 
restrictions encoded in the OntoSem lexicon and ontology 

• a statistically trained word sense disambiguation algorithm based on topic-oriented 
constraints 

• an algorithm for dynamic tightening and relaxation of constraints for cases of residual 
ambiguity and zero residual output candidates, respectively 

• a disambiguation algorithm based on estimating the similarity between pairs of concepts 
in the ontology by computing weighted distances in ontological space 

• an algorithm for unidirectional application of selection restrictions to process 
“unexpected” input words that are not in the system’s lexicon 

• an algorithm for deriving TMRs for fragments of input in cases of failure to produce 
TMRs for a complete sentence. 


Knowledge resources used in language understanding include grammars, lexicons, an 
ontology, a fact repository, micro theories of specific language phenomena and various 
specialized rule sets. The major stages in analyzing a dialog turn are 


• Preprocessing - text segmentation, processing of named entities, dates, numbers, 
abbreviations, punctuation, etc. and morphological analysis 

• Syntactic analysis - both constituent structure and dependency structure extraction 

• Basic semantic analysis - establishing “who did what to whom,” word sense 
disambiguation, semantic dependency detennination 

• Discourse analysis - concentrating at present on processing reference (including ellipsis) 
and speech acts. 
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As an illustration, consider the results of each of the above stages of analysis for the input 


“Come back to see me in six months.” 


Preprocessor results 

((COME V ((TENSE PRESENT)) NIL "Come") 

(BACKADVNIL NIL "back") 

(TO INF NIL NIL "to") 

(SEE V ((FORMINFINITIVE)) NIL "see") 

(ME N ((TYPE PRO)) NIL "me") 

(INPREP NIL NIL "in") 

(*MEASURED-OUANTITY* N NIL NIL "6 months") 
(*PERIOD* PUNCTNIL NIL 

Syntax: constituent structure 

(S (VP (VBP "Come") (ADVP (RB "back")) 

(S (VP (TO "to") (VP (VB "see") (NP (PRP "me")) 

(PP (IN "in ") (NP (NN "6 months "))))))) (PUNCT ")) 


Syntax: dependency structure (numbers refer to word numbers, starting with 0) 

((ADVMOD 0 1)(AUX3 2)(DOBJ 3 4)(POBJ 5 6)(PREP 3 5)(PUNCT 0 7)(XCOMP 0 3)) 

Basic text meaning representation: 

(RETURN-335 

((INSTANCE-OF (VALUE RETURN)) 

(WORD-NUM (VALUE 0)) 

(ROOT-WORDS (VALUE (COME))) 

(FROM-SENSE (VALUE COME-V7)) 
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(A GENT (VAL UE HUMAN-338)) 

(PURPOSE (VALUE CONSULT-339)))) 

(CONSULT-339 

((INSTANCE-OF (VALUE CONSULT)) 

(PURPOSE-OF (VALUE RETURN-335)) 

(WORD-NUM (VALUE 3)) 

(ROOT-WORDS (VALUE (*NULL* SEE))) 

(FROM-SENSE (VALUE SEE-V7)) 

(A GENT (VAL UE HUMAN-340)) 

(BENEFICIARY (VALUE HUMAN-339)))) 

(HUMAN-340 

((INSTANCE-OF (VALUEHUMAN)) 

(WORD-NUM(VALUE 4)) 

(ROOT-WORDS (VALUE (ME))) 

(FROM-SENSE (VALUE ME-N1)))) 

(SECOND-342 

((INSTANCE-OF (VALUE SECOND)) 

(WORD-NUM(VALUE 6)) 

(FROM-SENSE (VALUE *MEASURED-QUANTITY*-N1» 

(VALUE (VALUE 1.5778458E7))))) 

Discourse analysis adds 

(COREFER (VALUE HUMAN-338 HUMAN-FRO)) 

“FR” in the index of a concept instance means that it is a remembered instance stored in the 
agent’s memory of assertions, also called the fact repository. The above instance refers to 
“self.” 

(COREFER (VALUE HUMAN-340 HUMAN-FR12)) 

The above refers to the user. 

(REQUEST-ACTION-363 

((INSTANCE-OF (VALUE REQUEST-ACTION)) 
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(THEME (VALUE RETURN-335))))) 


The above is the speech act specification. 


The Operation of OntoSem is computationally expensive. In an attempt to make the cognitive 
agent’s language understanding more efficient - but also because this capability is something 
people use routinely - we attempt, whenever possible, to use analogical reasoning to obtain 
text meaning representations. Analogical reasoning is used for conversational formulas and 
frequent or common dialog turns. This latter capability is enabled in the MVP by the 
availability of the memory of assertions (the fact repository). 


As mentioned above, reasoning - using knowledge to make decisions - is a component of 
perception. Outside perception, the following reasoning processes are currently employed in 
MVP: 


• augmenting the memory (fact repository) with the new fact(s) 

• learning new lexicon entries and ontological concepts 

• augmenting/modifying beliefs about the knowledge and intentions of interlocutors 

• making decisions about what to do next: 

o running preference functions to select which goal and plan to pursue next or 
o finding analogous situations in hopes of using results of earlier reasoning 

• deriving the content to be generated into language. 
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For example, the goal be-healthy in the current version of the system has just two plans 
associated with it - see-md and do-nothing (the latter is selected in hopes that the condition 
will go away without treatment). Figure 3 illustrates the preference function for plan 
selection. 


BE-HEALTHY ;;a goal 

Define, retrieve or compute 
values for arguments of 
plan preference function 


(if 


(bind-variables 

(*health-attribute (get-attribute health-attribute *domain 1)) 

(‘severity (/ (round (* (- 1 *health-attribute) 100)) 100.0)) 

(*toleration (get-attribute ability-to-tolerate-symptoms *domain 0)) 
(*appointment (get-attribute appointment-time ‘domain -1)) 

(*fl (- *severity *toleration)) 

(‘previous-severity (get-attribute previous-severity *domain -1)) 
(*previous-time (get-attribute *domain -1))) 


(< (+ *previous-time 100) *time*) ;;don't run if already at MD office Plan Preference predicate 
(or for BE-HEALTHY 

(and (> *appointment 0) (>= ‘time* ‘appointment)) 

(and (< *appointment 0) (> *fl 0)) ;; *appointment < 0 means no appointment was set 
(and (< *appointment 0) (> *time-in-goal (* 6 30 60 60 24)) (> ‘severity 0)) 

(and (> ‘appointment 0) (<= ‘previous-severity 0.3) (>= ‘severity 0.5) (> *fl 0.1)) 

(and (> ‘appointment 0) (<= ‘previous-severity 0.5) (>= ‘severity 0.7) (> ‘fl 0.1)) 

(and (> ‘appointment 0) (<= ‘previous-severity 0.7) (>= ‘severity 0.9) (> *fl 0.1)))) 

then 

se&mt ;;a plan 

else 

do-nothing) ;;a plan 


Figure 3. Reasoning for selection of a plan for an instance of the be-healthy goal. 


The reasoning module in MVP concentrates on processing goals. The inventory of goals is at 
the moment limited to those relevant to the application. Goals are instantiated as a result of 
perception (“I feel pain”) or as a result of agent’s own reasoning (“I don’t know enough about 
the procedure recommended by the doctor to choose a course of action”). The choice of goals 
to pursue at a given time (some parallelism is simulated) depends on the extent and nature of 
the VP’s knowledge about the world, the contents of its memory of past events and a model 
of its personality traits, beliefs about self and other agents, genetic predispositions and 


362 


physical and mental states. Different VPs have not only different personality profiles and 
beliefs but also different ontology’s and fact repositories. The VP in the current version of the 
system is designed to reason by analogy: each goal is associated with a set of known plans. 
The current version does not include dynamic planning. 


The VP’s {what kind of actions - it has lots of other actions too} actions include: 


• learning new facts and remembering them 

• establishing reference relations between newly learned facts and stored facts 

• learning new concepts and lexical items 

• augmenting/modifying the agenda of goals and plans 

• generating language (dialog turns) 

• [simulated] physical action (e.g., showing up at the office or the lab, taking medicine, 
ingesting food/drink that affects health states, etc.) 

A typical situation in which the agent will leam by being told is as follows. The user may 
diagnose the VP with a disease about which it knows nothing other that what is diagnosed is 
a disease. On the one hand, the user may proceed to describe properties of the disease, in 
English; alternatively, the VP may ask the user questions about various properties of the 
disease. The VP knows what kinds of questions can be asked based on its ontological 
knowledge about diseases in general, and it decides whether or not to ask them based on its 
personality traits, current mental state, etc. During this interaction the VP will have to 
understand the text of the user’s dialog turn, extract from the TMR the filler or fillers of the 
property or properties in question and fill them in its ontology (or modify the existing filler if 
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there is one). The lexicon entry for the new lexical unit referring to the disease will contain in 
its semantic structure zone a direct pointer to the newly learned concept (this is the simplest 
option available). 


3. Comparisons with Other Systems 


This section presents a brief overview of some medically-oriented training systems with the 
purpose of positioning the MVP system on the map of VP-based training and medical 
simulation efforts. While the systems themselves and the domain they represent are not 
directly relevant to Companion development, the types of features that distinguish MVP from 
them are similar to the features that might distinguish MVP from some Companion systems. 


Cognitive modelling. 

As the description of MVP has shown, the environment as well as the patient itself model 
cognitive abilities and therefore can support non-trivial conversation, collaborative decision¬ 
making, realistic simulation, and so on. Other available medical training systems do not seek 
this level of cognitively-oriented functioning. A common type of medical simulation is 
realized in technical task trainers, which concentrate on developing motor skills and address 
cognitive skills only inasmuch as they are necessary for the user to understand a specific 
technical step, like how to insert a needle. A second type of simulation is realized in non-bio 
mechanistic mannikin trainers (e.g., “SimBaby” by Laerdal, Inc., and “The Human Patient 
Simulator” by Medical Education Technologies, Inc.), which focus on a narrow scope of 
acute physiological processes. A third type is not really a simulation but rather a branching 
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narrative scenario, organized as a decision tree, that depicts a specific medical case (e.g., 
MedCases, Inc.). In these, user options are restricted and responses are highly pre-scripted, 
being delivered through multiple-choice questions. Most importantly, in such systems patient 
outcomes are fully predetermined by the prefabricated scenario. A more sophisticated type of 
medical simulation is claimed to be implemented in the Sim-Patient developed by RTI, Inc., 
where acute traumatic patient scenarios are made available to a user; however, few details 
about this system are available as the data structures and content are proprietary. 


Several advanced scenario-based VP systems have been developed in the European project 
eVIP (http://www.virtualpatients.eu/). The systems differ as to delivery methods: e.g., the 
Web-SP system (Zary et al. 2006; Zary 2007) is designed for internet access but the 
underlying technology is more or less standard throughout. Thus, VPs are realized through 
collections of decision trees, with decisions expected to be made by human students on the 
basis of text presented in the prefabricated decision tree nodes that simulates patient 
examination and test results (both optionally with the help of pictures and photographs). No 
actual simulation of the patient’s physiology, character traits or cognitive abilities is involved. 
The centre of gravity in this project is in the area of interoperability, ease of authoring and 
curriculum acceptance of VPs (e.g., Ellaway 2009, Poulton 2009). 


The strategic decision to use pre-scripted scenarios and multiple-choice responses - and thus 
to bypass the complexities of physiological simulation, cognitive modelling and involved 
natural language processing led VP developers to concentrate on presentation issues. Videos 
of human actors, advanced graphics, including avatars, the incorporation of off-the-shelf 
speech recognition and synthesis software and visualization of results of medical tests (such 
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as X-rays and MRIs) have been prominent among the means of strengthening the 
verisimilitude of the human-computer interactive experience. Indeed, the latest VP 
environments (e.g., Courteille et al. 2008) show significant progress even when compared to 
quite recent ones (e.g., Chesher 2004). 


Authoring Agents. 

An unintended negative consequence of relying on prefabricated scenarios is the high cost of 
authoring patient instances, which limits the utility of scenario-oriented VPs. The scenario- 
oriented VP community is addressing this issue by attempting to share the burden across the 
development teams. This brings to the fore the issues of interoperability, standardization and 
efficient dissemination. However, the cost of developing a single training case remains quite 
high, requiring, according to some estimates, hundreds of person-hours (see Zary et al. 2006). 
Creating a training case for a VP that operates autonomously on the basis of a simulation of 
its physiological and pathological processes takes, as our experience in the MVP project has 
shown, just a few minutes. But the development of the prerequisites for quick patient 
authoring - a simulator and a model of disease progression for every disease (or, more 
precisely, group of diseases) - is, of course, an expensive undertaking. 


Is simulation needed? 

There is one notable medical training environment, CIRCSIM (Illinois Institute of 
Technology; Evens and Michael 2006), that originally relied on physiological simulation of 
the baroreceptor reflex but, over time, removed the simulation in favour of a handful of set 
scenarios that fulfilled all teaching requirements. As Michael and Rovick (1996) explain, 
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“The most effective teaching was being generated from the stored correct predictions for each 
procedure, not from the quantitative outputs generated by the model”. In MVP physiological 
simulation is a prerequisite for the VP’s ability to reason independently and initiate behavior. 
Removing this option would seriously compromise the richness of the repertoire of VP 
actions and, therefore, be detrimental to its verisimilitude level. 


Knowledge-based vs. probabilistic simulation. 

Walton Sumner and his associates (Sumner and Hagen 2006, Sumner et al. 1996, Marek et 
al. 1996) developed a virtual patient system for the purpose of advancing medical 
certification procedures of the American Board of Internal Medicine. This system goes 
beyond the level of multiple-choice questions and relies on simulating a patient with possibly 
multiple co-existing conditions and allowing the examinee to intervene. The system addresses 
the issue of automatically creating instances of VPs by starting with a selected disease (or 
diseases) and probabilistically creating a hypothesized medical history for each patient 
instance based on knowledge about the incidence of this disease (or diseases) in the 
population. This emphasis on generating differentiated patient histories is due to the 
perceived need for providing a secure testing environment in which patient instances are not 
reused. The probabilistic nature of much of the operation of this system suggests the use of 
Bayesian networks as the underlying representational mechanism, which adds complexity to 
both knowledge acquisition and processing. In MVP there is no need for probabilistic 
reasoning because VP instances are not probabilistic; they are authored by the teachers and 
represent a specific, not probabilistic set of choices of disease progression and personality 
traits. If the teacher needs to present a different set of disease progression and intervention 
outcomes, he or she does this by authoring a different deterministic VP instance. What is not 
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deterministic in the environment is the behavior of the human user, and there is no reason to 


develop a probabilistic model of this behavior. 


Non-VP medical agent-based systems 

One example of the use of the concept of artificial intelligent agents in the medical domain in 
the “anthropic agency” approach (Amigoni et al. 2003), implemented as a multiagent 
environment for modelling and regulating physiological phenomena, specifically, the insulin 
and glucose levels in diabetes patients. This system relies on what the authors call the 
anthropic agency architecture, “a powerful paradigm to develop control systems for 
physiological processes shaped as multiagent systems.” (p. 310). The architecture consists of 
the traditional steps of perception (called “knowledge extraction”, implemented using a team 
of identical “extractor agents”), reasoning (“decision making”, implemented using a team of 
identical “decisional agents”) and action (“plan generation” with their team of “actuator 
agents”). All the agents in this system are what can be called low-level agents, specialized 
computer programs (in contrast to high-level agents that simulate a broad spectrum of 
human-like behaviors). The physiological model has a narrowly directed coverage, no 
cognitive abilities are simulated for the virtual patient and no network of high-level agents 
simulating human capabilities is introduced. In general, the complexity of the domain 
knowledge is not the main focus of this work. 
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Discussion 


This chapter has described a task-oriented conversational Companion. However, the same 
types of capabilities that permit a VP to carry out the kinds of behavior described above can 
facilitate the development of a conversational Companion that provides its user with a rich, 
multi-faceted virtual experience. In fact, as detailed in the table below, the MVP already 
includes functionalities that directly address the desiderata for conversational Companions 
suggested by Pullman (this volume). 


Pulman’s Desideratum 

Relevant MVP features 

A Companion should have 

The VP uses recorded plans that include actions in which 

intentions toward us 

the user (the MD) is the agent; in other words, the VP is 

programmed to delegate some parts of the process of 

attaining the goal of “be-healthy” to the MD 

A Companion should recognize us 

The VP maintains a profile of its user in the form of a 

as individuals 

marked instance of the ontological concept HUMAN and 

associates beliefs, history of interaction and trust 

parameters with a particular user 

A Companion’s behavior should be 

An objective of the MVP environment is to expose the 

predictable 

student to as many realistic kinds of patient as possible; this 

objective does not dovetail with that of a Companion 

because the user of the latter will benefit from building a 

mental model of a specific Companion, which helps to 

suspend the user’s disbelief and facilitate the acceptance of 

the Companion by the user; 
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still, if exposed to the same VP a second time, the medical 

student will be able to predict the VP’s behavior. 

Our behavior should be predictable 

The VP is programmed to expect that the interaction with 

to a Companion 

the user is predictably limited to diagnosis and treatment of 

a medical condition 

Companions should be 

The only effort that the user may expend is to teach the VP 

independent they should not 

about its condition, which can be considered an integral 

require (much) effort from us 

part of diagnosis and treatment. However, teaching the 

Companion, though an effort, may be considered a positive 

task for the user, as this leads to closer ties between the user 

and the Companion and helps the user feel needed, which is 

an important component of feeling happy (which is a stated 

goal for conversational Companions). 


Finally, to answer the natural question, MVP has not yet been subject to formal evaluation 
but experts in the field of medical education and several groups of medical students have 
resoundingly approved the working prototype. While they stress the potential impact of MVP 
on the field of medical education, we view it far more broadly, understanding that all 
progress on the core capabilities of human-like artificial agents can extend to any domain. 
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Living with robots: ethical tradeoffs in eldercare - Noel Sharkey and 
Amanda Sharkey 


Abstract 

We discuss some of the research and ideas for developing robot carers and 
Companions for people with aging brains. We speculate a little about a 
near-future when it may be possible to keep people at home for longer in 
the almost exclusive care of robots and smart homes. We examine the 
benefits of robot care and Companionship and at the same time raise key 
ethical questions and concerns. We point to a series of trade-offs between 
the unethical and the beneficial that must be considered before robot 
care/Companionship becomes commonplace. 

Who would leave their granny or aging parents in the exclusive care of robots? When one of 
us asked a family audience of around 500, all the children eagerly raised their hands while the 
adults kept theirs down. The adults shook their heads and looked with surprise at the children 
because Noel had just been pointing out the potential downside of the extremes of robot elder 
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care - loss of human contact. Worryingly, this is the generation that will decide on our future 
as we grow old and their attitude and approach to technology may be very different from 
ours. 

The possibility of being cared for exclusively by robots is no longer science fiction. Japan 
has, for some time, had an eye to using machines to fill the care gap both in terms of labour 
shortages and economic considerations. Because of a low birth rate, no immigration, and long 
life expectancy, their population is aging rapidly, with 22% over the age of 65 in 2009, 
predicted to rise to 34% by 2035. 76 The worry is that there will not be enough young to look 
after the old and so it is part of the Japanese government’s health policy to use robots for 
care. In March 2009, a Japanese Ministry of Trade and Industry official, Motoki Korenaga, 
told Agence France-Presse that, “Japan wants to become an advanced country in the area of 
addressing the aging society with the use of robots." 77 He spoke of plans to prepare safety 
rules soon for robot nurses that are expected to serve elderly needs within the next five years. 

Japan is already en route to deliver robot assisted care with examples such as the Secom “My 
Spoon” automatic feeding robot, the Sanyo electric bathtub robot that automatically washes 
and rinses, Mitsibushi’s Wakamura robot for monitoring, delivering messages and reminding 
about taking medicine and Riken’s RI-MAN robot that can pick up and carry humans and 
follow simple voice commands and even answer them. The idea is to follow these by 
developing robots that can do many of the household chores that a visiting helper is now 
required for. A short Google search will throw up many. 

Other countries may well follow suit. Europe and the United States are facing similar aging 
population problems over a slightly longer time scale. In 2009, it is estimated that 16.2% of 
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the population in the UK is aged 65 or older (CIA World Factbook). Spain and Italy are the 
oldest in Europe with 18.1%, and 20.2% respectively (CIA World Factbook) and these 
figures are increasing sharply. In the UK, the fastest growing age group is made up of those 
aged 80 years and over who now constitute 4.5% of the population. In the US, 12.8% are 
over the age of 65, expected to rise to around 20% by 2030. It seems likely that Europe and 
the US may follow the Japanese lead into robot care. 

Work has already begun in the US. Pearl, developed at CMU (Pollack et al, 2002), is a 
nursebot that reminds seniors about routine activities (e.g. eating, drinking, taking medicine 
and using the bathroom/toilet), and that can guide them through their environment. RP-7 is a 
teleoperated robot that is used to facilitate doctor-patient interactions without the doctor 
leaving her office. It has been used at the Silverado Senior Living Apsen Park (Winner, 

2007). The University of Massachusetts, Amherst robot, uBot5 has manipulators that enable 
it to perfonn a number of tasks such as picking up and moving objects around. It has been 
suggested as a means of doing house calls to check for signs of a stroke and could be used to 
monitor an old person for signs of a fall. 

There is a massive drive to keep the elderly out of expensive residential care homes for as 
long as possible. Apart from the cost, in the UK there has been considerable concern about 
the conditions of many care homes. According to a UK Ministry of Justice and Department of 
Health report (2007), as many as 50,000 older people with dementia in hospitals or care 
homes could be being deprived of their liberty without access to an appeal process. There 
have been reports of malnourishment, abuse and poor care. Following a disturbing expose on 
the Panorama (BBC, April, 2009) in which undercover reporters found work in care homes, 
Michelle Mitchell, Charity Director for ‘Age Concern and Help the Aged’, said, “The current 
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care and support system needs straightening out, starting with a new national entitlement to 
services that is designed around people’s needs.” 

At present most of the robot carers have been tried and tested in residential homes for the 
elderly and their use is still quite novel. They are expensive at present but the costs are falling 
all the time. Robots were 80% cheaper in 2006 than they were in 1996 (World Robotics 
report 2007) and so robot care may become economically viable over the next 10 years or so, 
and cheaper than human care. The long term plan will be to have these care robots in the 
homes of the elderly to help keep them out of care homes for longer. 

Assistive care with robot technology has the potential to allow greater independence for the 
elderly with dementia or other aging brain symptoms. However, such care may could come 
with a moral cost to society (Sharkey, 2008). Our aim here is to throw up some of the ethical 
questions that will be raised as the robotics field progresses sufficiently to allow near¬ 
exclusive care of the elderly by robots and machines. We present both the benefits and 
ethical issues as tradeoffs that need to be balanced in favour of the well-being of those being 
cared for. 


Safety in the home 

For a robot to be useful in the home care of the future, it will have to operate safely on its 
own and keep its elderly charge from physical harm. There are lessons that can be learned 
from child care robots currently on the market. Some of these such as the PaPeRo by NEC 
can keep track of people by attaching radio frequency identification (RFID) tags to them. 
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This type of robot also enables remote monitoring through camera “eyes”. A carer can watch 
on a window on their computer screen or on their mobile phone. They can also speak 
through the robot and remote control it or give it instructions to perform a task. 

Robots could be used for health monitoring in a number of ways such as close contact 
measurements of temperature, respiration and pulse. In the hi-tech retirement home run by 
Matsushita Electrics, robot teddy bears watch over elderly residents; monitoring their 
response time to spoken questions, and recording how long they take to perfonn certain tasks. 
They can alert staff to unexpected changes. This is an area that, once developed, could have a 
significant impact on elder care. 

We are likely to see the integration of robots with other home sensing and monitoring 
systems. There is considerable research on the development of smart sensing homes for the 
frail elderly. These can monitor a range of potentially dangerous activities such as leaving on 
water taps or cookers They can monitor a senior leaving her bed at night and wandering. 
They can prompt the person with a voice to remind them to go to the toilet and switch the 
toilet light on for them (Orpwood et al, 2008). Vision systems can detect a fall and other 
sensors can determine if assistance is required (Fu et al, 2008). 

A robot could greatly extend the flexibility of such “smart homes” as better tracking facilities 
become available for robots. For example, Lopes et al, (2009) have developed a method that 
allows tracking of people in a range of environments and lighting conditions without the use 
of sensor beacons. This means that the robot will be able to follow a senior outside and alert 
carers of her location or encourage and guide her back into the home. The ethical danger here 
is that this could turn into the equivalent of imprisonment in the home without trial. 
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If robots are to be used to reduce human care, they will need to function more as autonomous 
supervisors. A simple extension would be to allow home customisation so that the robot can 
recognise danger areas such as drawers containing potentially dangerous objects. As in the 
smart sensing home, the robot could make a first pass at warning a senior to stop doing or 
engaging in a potentially dangerous activity. But there may be ethical problems here. If a 
robot can predict a dangerous situation it could be programmed to autonomously take steps to 
prevent it, e.g. taking matches out of the hands of a senior, getting between them and a 
danger area or even restraining them (gently) to prevent them from carrying out a dangerous 
action. However, restraining a person to avoid harm could be a slippery slope towards 
authoritarian robotics. 

It would not be difficult to construct scenarios where it would be hard to deny such robot 
action. For example, if a old person was about walk onto the road into heavy oncoming traffic 
and a robot could stop her, should it not do so? The problem here is in trusting the robot’s 
classification and sensing systems to determine what is a dangerous activity. As an extreme 
case, imagine an elder having high calorie foods taken from her because the robot was 
preventing her from becoming harmfully obese. How would the law view a carer using a 
remote controlled robot to restrain an elder or take objects of potential harm from her? There 
are many discussions to be had over the extremes of robot constraint and where to draw the 
line. 

Another ethical issue arising from the safety and monitoring properties of care robots is 
concerned with the right of all people to privacy. A person with Alzheimer’s may soon forget 
about robot installations and perform acts or say things thinking that she is in the privacy of 
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her own home or thinking that she is alone with her robot friend. With the massive memory 
hard drives available today, it would be possible to record the entire remainder of an old 
person’s life. While the idea of preserving the memories of one’s elderly parent may seem 
attractive, it might not be something that they would consent to if able. Is this kind of Big 
Brother invigilation acceptable? A perhaps more important question here is, who should have 
access to the recordings? Would you want your children to know everything you said about 
them to an artefact? 


Robot Companions and traditional dolls in therapeutic applications 

While robots might eventually be beneficial in keeping old people out of care homes for 
longer, we have a duty of care to the elderly not to leave them in the exclusive company of 
machines. The elderly need the human contact that is often only provided by human carers 
and people performing menial tasks for them. There is a foreseeable future in which 
affordable robots may be able to cater for all the needs of seniors with dementia. Will the 
vulnerable elderly then be left without Companionship? 

One take on this problem, particularly in Japan, is to move towards the development of robot 
Companions and robot pets. These are being touted as a solution to the contact problem - 
devices that can offer Companionship, entertainment, and human-like support to the elderly 
at home alone. There are a number of robot pets that have been mooted as possible 
Companions for the elderly, although usually not explicitly designed as such. They are 
designed as sophisticated toys. We describe four of these. 
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Paro, a fur covered robotic seal, was designed for therapeutic uses with the elderly. 
Developed by AIST, it responds to petting by moving its tail and opening and closing its 
eyes. It reacts to sounds and can leam to respond to its name. It makes seal-like sounds, and 
is active in the day, and prefers to “sleep” at night. It can detect light and dark by means of a 
light sensor, and recognise when it is being held, stroked, or hit, by means of posture and 
tactile sensors. 

Sony’s AIBO robotic dog, developed as an “entertainment robot”, has also been used in 
robot Companions research. It has a metallic dog-like fonn, and can walk, or chase a ball. It 
has sensors that can detect distance, acceleration, sound, vibration and pressure. It can 
express six “emotions” (happiness, anger, fear, sadness, surprise and dislike), by means of its 
tail, body movements, and the colour and shape of its eyes. More recent versions can 
recognise voice commands, and the robot exhibits slightly different behavior depending on 
the interactions it has experienced. 

Other similar robots include NeCoRo (OMRON), a robotic cat covered in synthetic fur. Its 
behavior depends on the history of its interactions and it can ‘learn’ to recognise its name. 
Stroking and hugging it elicits positive responses while treating it roughly elicits angry 
behavior. Like Paro, it has internal rhythms that lead it to sleep, or seek to be cuddled. 

My Real Baby (iRobot) is described as an “interactive emotionally responsive doll”. It can 
make different expressions, blink, and suck its thumb or bottle. It changes its expressed 
emotion depending on how it is treated - so that when fed and rocked for instance, it behaves 
as though it were sleepy. When unfed, it gets hungry and cries. It also makes a number of 
random sounds and words, which become more like sentences the longer it is played with. 
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Research shows that old people can benefit from interacting with such robots. Often the gains 
shown are similar to those obtained from animal assisted therapy in which visits from a pet 
animal to residents of long term care facilities can reduce loneliness (Banks and Banks 2002; 
2005). Kanamori et al (2002) showed various improvements in elderly persons who 
interacted regularly with a Sony AIBO robotic dog - their loneliness scores were reduced, 
and their Quality of Life assessment scores improved. Banks, Willoughby and Banks (2008) 
even found that old people in long term care facilities benefited as much from interacting 
with an AIBO robotic dog as from interacting with a real dog; both resulting in reductions in 
their reported levels of loneliness. Elderly dementia patients have also shown positive 
outcomes, including increased communication, as a result of sessions with an AIBO (Tamura 
et al, 2004). 

Positive results have also been obtained with “doll therapy” where dolls are ‘introduced’ to 
clients to stimulate memories of a rewarding life role, especially that of a parent, and to act as 
a focus for reminiscence and conversation. “ (Cayton, 2006). This is closely related to robot 
Companions research. The “Someone to Care For“ doll is made especially for the elderly and 
the manufacturers claim that, “These beautiful dolls offer comfort, care and happiness to 
senior citizens, especially people living with Alzheimer’s disease. Show someone how much 
you care with a Someone To Care For doll.” 

As in the robot Companions research, the results from a number of studies suggests that doll 
therapy can be beneficial in reducing negative behavior and aggression and promoting a 
positive attitude (Ellingford et al ., 2007; James et al ., 2006). Some of the care workers who 
were interviewed felt that the dolls helped with communication - residents with dolls 
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appeared “much calmer and more approachable” (Mackenzie, et al. 2006). It seems that there 
are also obvious sex differences in doll therapy. When given a choice about doll therapy, 
males tend to stick to their stereotypical role of not wanting to “play” with dolls. Robots 
might suit their dignity more. In fact, it has been suggested that the elderly are generally 
more comfortable playing at family scenes with robotic dolls than with traditional ones: 
“Seniors felt social ‘permission’ with the robots presented as a highly valued and ‘grown up’ 
activity. ” (Turkle et al, 2006 pg 354). 

There has so far been insufficient research in on the relative merits of dolls and robots. In one 
study, Libin and Cohen-Mansfield (2004) compared the benefits of a robotic cat (NeCoRo) 
with a traditional fluffy toy cat (plush) as an interventions for seniors with dementia. The 
plush toy cat was entirely passive and had to be manipulated by the seniors themselves. The 
NeCoRo cat, described as “an emotional communication robot” (Ogata et al., 2000; Shibata 
et al., 1999), had a variety of sensors to enable interactive responses to stroking etc. It could 
produce a meow, a purr or a hiss and could perform actions such as stretching its paws, 
wagging its tail, opening and closing its eyes, lying down and sitting or turning its head and 
spreading its ears. Despite these differences, the therapeutic use of both cats lead to similar 
results. They both resulted in decreased agitation together with increased pleasure and 
interest. Only nine clients were used in this study and so the jury is still out on the different 
merits of robots and dolls. 


Deceit, illusion and infantilization 
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An elderly person with dementia entertaining a relationship with a robot may not be in a 
position to distinguish this from a relationship with a socially and emotionally competent 
being. Sparrow pointed out that the relationships of seniors with robot pets, “are predicated 
on mistaking, at a conscious or unconscious level, the robot for a real animal. For an 
individual to benefit significantly from ownership of a robot pet they must systematically 
delude themselves regarding the real nature of their relation with the animal. It requires 
sentimentality of a morally deplorable sort. Indulging in such sentimentality violates a (weak) 
duty that we have to ourselves to apprehend the world accurately. The design and 
manufacture of these robots is unethical in so far as it presupposes or encourages this” 
(Sparrow, 2002). 

These concerns are mirrored in some of the doll therapy literature. Studies have shown that 
clients with dementia engaged in doll therapy tend to believe that their dolls are real babies. 
Women, in particular may believe that they have reverted to their early 20s when they were 
mothering young children (James, et al. 2006). When Mackenzie, et al. (2006) questioned the 
care workers in homes where doll therapy had been tried, they discovered that some residents 
had been trying to feed their dolls [unsuccessfully]. Some over-invested in their doll and 
would put its interests before their own as one would with a real baby. James et al (2006) had 
similar results with the dolls being perceived to be babies and residents wanting to bathe 
them or give up their own bed for the ‘baby’ to have a good night’s sleep. 

Some dementia professionals have been critical of the use of dolls on ethical grounds. In the 
‘treatment’ of dementia, the use of doll therapy is predicated on the notion that those with 
dementia are going through a second childhood (e.g. Reisberg et al., 2002). Critics such as 
Cayton (2006) see the notion of second childhood as dispiriting and as encouraging a rigid, 
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authoritarian, deficit-based approach to care. It is what Kitwood (1999) describes as a 
malignant way of caring that leads to disempowerment, disparagement and infantilization. It 
leaves little room for the recognition of the elders’ lifetime experiences that separate them 
from children (Salari, 2002). Both Mackenzie, et al. (2006) and James et al. (2006) report 
that some carers, visiting relatives and fellow residents saw the doll therapy as demeaning, 
patronising and inappropriate. 

In discussing the ethics of doll therapy, Cayton (2006) expresses doubts that the elders are 
engaged in conscious play in the way children are: 

“When children play make-believe, ‘let’s pretend’ games they absolutely 
know it is pretend. That is why it is safe and liberating. It enables children to 
try out roles, take risks and explore relationships. Real play is a conscious 
activity. Ask a child who is playing with a doll what they are doing and they 
may tell you matter-of-factly that they are going to the shops or that the doll 
is sick but they will also tell you that they are playing. A person with 
dementia does not do that I think ’’.(Cayton, 2006 p283) 

We could still say that if the old person saw a robot or doll as real it is still not a deceit 
because it is natural anthropomorphism (Sharkey and Sharkey, 2006) combined with an aging 
brain that leaves them unable to see the reality behind the illusion. Where the deceit arises is 
in giving someone an artefact knowing that they will believe it to be alive. This point is 
reinforced by Cayton who clearly believes doll therapy to be deceit: “The ‘therapist’ knows 
this is a doll. Do they share that fact with the client or do they pretend it is a baby? “ He asks 
therapists, “are we colluding with a misunderstanding or are we asking people with dementia 
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to collude with us?“ This issue of deceit raises difficult ethical questions. One the one hand, 
if you believe that all deceit is morally wrong, then you will categorically oppose the use of 
care robots. On the other hand, you may believe that the beneficial consequences of colluding 
in the deceit of an elder is worth a little untruth: “they are never going to know anyway and 
so what is the harm.” But even then we still need to consider whether the benefit to the elders 
outweighs a potential erosion of society’s moral standards. 


The human touch 

The biggest concern we have about home robot care is that it may replace human contact 
altogether. With very advanced smart sensing systems and robots that can lift and carry, bathe 
and feed as well as keep safe, there will be less need for care visits - the whole point of using 
the robots is because there will be less carers available as the population ages. This is bad 
news for many elderly people for whom visiting carers are the only human Companionship 
they have on a daily basis. According to a report from the charity Help the Aged (2008), 

17% of older people have less than weekly contact with family, friends and neighbours 
andl 1% have less than monthly contact. 

It may be that one day, in the distant future, there will be humanoid robots that appear to be 
human, at least to people in the throes of severe dementia. For now and for some time into the 
future, the natural language processing and speech recognition facilities available to robots 
would not make for a great surrogate human Companion. A Reuters’ (2007) report told of the 
failure of the ifbot to hold the attention of the elderly for long. It is marketed as a robot that 
can converse, sing, express emotions and give trivia quizzes to help seniors with their mental 
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agility. But the residents of one facility in Kyoto lost interest after only a month and then the 
robot was ignored for the next two years. The director of the facility said that stuffed animals 
were more popular. 

It is the natural right of all individuals to have contact with other humans and to be allowed to 
socialise. As Boas (1998) points out, in discussing the effect of new therapies for people with 
aging brains, “.. .what simulates them, gives a lift to their spirits, is the human interaction, the 
Companionship of fellow human beings.” And having a good social network helps to protect 
against declining cognitive functions and incidence of dementia (Crooks et al. 2008; Bennett 
et al. 2006). Society has a duty of care and a moral responsibility to do its best to ensure the 
emotional and psychological wellbeing of all of its citizens regardless of their age. We must 
not allow Companion carers to be used for alleviating our guilt about the isolation and 
loneliness suffered by many elderly people. 

As we suggested earlier, safety measures and over-supervision could potentially turn the 
home into a prison. In 2008 already 12% of older people in the UK (over 1.1 million) feel 
trapped in their own home, 7% (nearly 700,000) don’t go out more than once a week and 3% 
of older people never go out. Safe robot care could exacerbate this situation if precautions are 
not taken to ensure that regulations and guidelines are set up to avoid social exclusion. 

On the brighter side, robots could be used to facilitate an elderly person’s social life. In the 
mid-term future, robots should be able to navigate seniors on a walk around the 
neighbourhood and even to take them shopping. The presence of a robot with facilities to 
interact with people could be used to encourage more engagement in the community with the 
senior. With much of the dull and dirty work and supervision being taken over by robots, a 
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smaller number of human carers could work to ensure that the elderly have the possibility of 
a social life. James et al (2006) observed groups of “mums” in their “doll therapy” study that 
voluntarily got together every day with their dolls. Similarly, human carers could facilitate 
interest groups of clients with robot gizmos. These could be socially rewarding without need 
for detailed conversation. 

When Wada and Shibata (2006) videoed interactions between a Paro robot seal and a group 
of elderly care home residents, they found that the social interactions between the residents 
themselves had increased and physiological indicators showed reduced stress levels. It 
seemed that Paro encouraged positive communication, and resulted in a reduction of the 
“backbiting” that had previously characterised some of their interactions. One 75 year old 
female resident greatly increased her interaction with fellow residents. A friend of hers 
commented that she had been taciturn before Paro’s arrival, but now she was more cheerful 
and talkative. 

A robot that facilitates conversation may also be an attractor for visitors. It gives a focus of 
attention for chat. Children may want to play with the robot and have fun with granddad’s big 
toy. Kanamori et al (2002) report the case of an 84-year-old man with cerebral apoplexy 
sequelae. He talked much more to his children after the introduction of an AIBO robot dog. It 
gave both him and them a focussed object to talk about. 

Conclusions 

We have speculated about a near-future in which it may be possible to keep the elderly with 
dementia out of residential care for longer in the almost exclusive care of robots within a 
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smart sensing environment. We looked at some of the benefits both in terms of keeping the 
elders safely in their own homes, tending to their physical needs and in terms of caring for 
their emotional and psychological well being. With the population aging in the industrialised 
nations, there eventually will be a greater shortage of care staff than there is at present. In the 
UK there has been considerable concern about the conditions of many care homes and this 
concern has been reflected in many other countries. Such conditions can lead to a more 
rapidly progressive loss of mental faculties. 

The new technology could help to keep the elderly safe from physical harm within their own 
homes and help deter them from wandering into danger outside. The ethical tradeoffs we 
raised were about the use of restraint that could restrict liberties, virtual house 
arrest/imprisonment and depriving seniors of their right to privacy. We need to design the 
systems around the needs of individuals rather than force seniors to adapt to the technology; 
we need to ensure a fair and equitable tradeoff between their rights and freedoms and their 
safety. We must ensure that a line is drawn between care and oppression. 

In section 2, we discussed the potential psychological and emotional benefits of robot 
Companions. Studies on the effects of elder interaction with robot pets and dolls almost 
uniformly show positive results. They appear to reduce loneliness, negative behavior, 
agitation, stress levels and aggression. The ethical tradeoffs are in terms of dignity and deceit: 
elders may think that they are dealing with real babies or forming real relationships with 
robots that, in reality, can’t love them back. 

Research studies in care homes show that some elders even put the “interests” of their 
artefacts ahead of their own well-being. Fellow residents, relatives and care workers said that 
emotionally engaging with the artefacts was demeaning, patronising and inappropriate. 
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Professionals in the field have voiced concerns about the whole idea of treating dementia as 
leading to a second childhood. They proclaim that it leads to disempowerment, 
disparagement and infantilization. We must consider these issues before handing out robot 
Companions wholesale and ensure that the benefit to the elders outweighs a potential erosion 
of society’s moral standards. 

Our biggest concern about near-exclusive care of the elderly by machines is that it could 
deprive them of human contact and lead them into an isolated existence to finish their days in 
loneliness. This is a worst case scenario but not inevitable if we get our priorities right. 
Facilitating social interaction should figure highly in the initial design of robot 
care/Companion systems. The senior may not have had many interests to talk about before 
the new found robot friend arrived and robots are still novel enough to be of interest to 
younger people. A robot may assist in providing the client with more social opportunities by, 
for example, by navigating them around the neighbourhood or to shopping areas. If designed 
with interacting with others in mind, a robot could gain attention from others and provide a 
focussed topic of conversation or other interaction. This may also attract more family visitors 
with children to their homes. 

As a society we have a moral duty of care to all of our citizens regardless of their age. We 
must protect the weak, sick and vulnerable both physically and psychologically. We have 
seen the benefits of robot care and Companionship and discussed the ethical tradeoffs. Our 
collective responsibility is to ensure that the tradeoffs are weighed in favour of protecting the 
rights and dignity of our dependant seniors. The types of robot Companion carers that we 
have discussed here are currently little more than toys and we must not allow them to serve 
only to alleviate society’s guilt about elder isolation. 
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Section Six - Afterward 


388 



Summary and discussion of the issues - Malcom Peltu and Yorick Wilks 


Abstract 

The COMPANIONS project, which inspired this book, is studying conversational software- 
based artificial agents that will get to know their owners over a substantial period. These 
could be developed to advise, comfort and carry out a wide range of functions to support 
diverse personal and social needs, such as to be ‘artificial Companions’ for the elderly, 
helping their owners to leam, or assisting to sustain their owners’ fitness and health. This 
chapter summarizes the main issues raised in the workshop that gave rise to this book. Most 
direct quotes from participants in this chapter come from their own chapters. Appendix 1 
contains examples of current artificial Companions and related research projects mentioned at 
the workshop. 


Overview: the nature and significance of artificial Companions 

What is an artificial Companion? 78 

Artificial Companions (ACs) are typically intelligent cognitive ‘ agents ', implemented in 
software or a physical embodiment such as a robot. They can stay with their ‘owner’ for long 
periods of time, learning to ‘know’ their owner’s preferences, habits and wishes. An AC 
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could enter a close relationship with its owner by chatting to, advising, informing, 
entertaining, comforting, assisting with tasks and otherwise supporting her or him. In doing 
this, the Companion should make no technical demands on the user. 


This view of artificial Companions, also known as ‘ digital Companions ’ or ‘embodied 
conversational agents (ECAs) ’ 79 , overlaps with a number of similar research conceptions 
crossing a variety of academic disciplines, such as ‘affective computing f Picard 1997) 80 , 
‘emotion-oriented systems’ 81 , ‘relational artefacts ’ (Turkle 2006) and ‘humanoid robotics ,82 . 
The COMPANIONS project itself emphasizes language-centered software agents, rather than 
robotics. 


These kinds of artificial Companions could alter the way we think about the relationships of 
people to computers, the Internet and other information and communication technologies 
(ICTs). These changes could also affect relations, both individual and social, between human 
beings who own Companions. 


Figure 1 illustrates a few of the many artificial Companions mentioned in the book and at the 
original workshop. The core areas of interest in these discussions were the more advanced 
Al-based software developments that move toward creating new forms of more intimate 
‘emotional ’ relationships between people and computers. 
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Figure 1. Examples of current artificial Companions* 


Type 

'Intelligent’ toy, ‘pet’ 
and domestic robots, 
from which valuable 
knowledge on users’ 
emotional reactions to 
ACs can be drawn 


Software-based 

Companions 


Humanoid and other 
forms of advanced 
robotics 


Examples 

Sony Aibo robot dog 

Furby, an owl-like robot that appears to learn English 

Paro, a seal-like robot that seeks to create an emotional attachment in the 
owner. 

Primo Puel talking doll 

Tamagotchi, virtual creatures that can ‘grow’ from being a ‘child’ to 
‘healthy adult’, but ‘die’ if they are neglected. 

Roomba vacuum cleaner robot 

BASIC (Believable Adaptable Socially Intelligent Character) 

Beating the Blues computer-based therapy 
Greta, a virtual 3D embodied conversational agent 

Laura fitness health adviser 

Ecobot energetically autonomous robots 

Kismet ‘sociable machine’ 

KASPAR, a child-like robot for studying human-robot interaction 

Pearl nursebot 

Radar learning robot 
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Congenial support for the vulnerable: the COMPANIONS approach 


The European Commission’s COMPANIONS 83 project—which was the prime inspiration for 
the workshop preceding this book—adopts a widely held perception of ACs as being most 
suitable as a friendly ’ and socially-valuable support for a wide range of targeted groups, like 
the elderly, fitness-conscious, young or disabled. It is particularly interested in creating 
software Companions that could act as an enhanced interface to the future Internet by 
providing new and much more personal ways of dealing with the ‘ overload ’ of information 
opened up by the World Wide Web and other ICTs. In particular, it is exploring the 
management of personal information about the owner’s life and memories. 


Such agents would be integrated as the human interface to the development of a ‘more 
intelligent Web\ such as the proposed ''Semantic Web ,S4 . These agents would be able to 
exploit the richer and more rigorous linking of data, which Shadbolt et al. (2006) argue is a 
key goal of the envisaged progression from the World Wide Web of documents to the 
Semantic Web of data. This will enable Web content to be accessed directly by users or 
software ‘ agents' through an understanding of the meaning of the content of texts on the Web 
(see also Wilks 2006: 16-17). Artificial Companions will be able to draw on such 
developments to support their owners 85 . 


COMPANIONS is contributing to, and draws on, wider AC and related developments. It is 
developing demonstrators consisting of ‘persistent ’ (long-term) personalized agents that will 
engage in ongoing conversations with their users to build a narrative about that person’s life 
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and needs over a long period of time. These are targeting two main areas of support. One is a 
Senior Companion 86 agent that will communicate with its user by adapting to his or her voice, 
needs and interests in order to provide company for the lonely and to help access infonnation 
and services, including how to react to emergencies. The other is a Health and Fitness 
Companion 87 that will support healthy eating habits and fitness activities by maintaining 
records of its user’s health-related, eating and exercise information. These are typical of the 
AC applications envisaged by many forum participants. 


The project is concentrating on three key AC technologies in building these demonstrators: 
memories for life and identity (e.g. Wilks 2006) 88 ; natural language processing (NLP) and 
speech technology 89 ; and agents and the Semantic Web (Wilks forthcoming). The 
demonstrators will deploy multimodal methods of communication, including NLP dialogues, 
speech, advanced visual technologies 90 (e.g. tracking eye movements 91 ), sophisticated facial 
expressions and other non-verbal cues (e.g. Andre et al 2004; Andre 2007; Pelachaud 2007), 
touch screens and sensors. 


Roots and branches of the artificial Companions family 

Two main streams of research were identified as the prime sources of current artificial 
Companion work: Artificial Intelligence (AI) and human-computer interaction (HCI). Taylor 
and Swan (this volume) summarize in their e-Horizon forum position paper how these 
streams have converged to influence AC design and development. 
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They point out that AI and associated robotics approaches have undergone ‘significant and 
dramatic changes since the 1950s ’ (e.g. see Turing 1950; Newell and Simon 1963; Sloman 
1978; Weizenbaum 1984; Crevier 1993; Papert 1993; Whitby 1996; Brooks 2002; Boden 
2006) 92 . They comment that the traditional ‘top-down, brute force ’ AI approach has been, 
over time, largely replaced by an AI ‘which envisages learning as something that evolves 
from the ground up 


In this evolution, according to Taylor and Swan, the AI focus has moved away from building 
autonomous machines. Instead, they say: ‘No longer is it exclusively assumed that the sort of 
intelligence to be attained in machines should simulate human intelligence. ’ They identify 
Suchman’s (1987; 2006) critique of some assumptions implicit in AI as a pivotal 
development opening new possibilities for fresh thinking on framing ideas about machine and 
human intelligence. Suchman’s work has contributed to a growing recognition in HCI of the 
‘situated’ character of human action: the way that even planned behavior is contingent on the 
contexts we find ourselves in (see also, for example, Avgerou 2004; Jacko and Sears 2003; 
Latour 1993). Even more significantly, they say, are Suchman’s insights into the 
‘constitutive ’ nature of the human-machine intersection: how human-computer actions and 
interactions can reconfigure the ways in which humans and machines are understood. Taylor 
and Swan note that this is also related to the re-casting of distinctions made between humans 
(and animals) and ‘things for instance through developments such as Actor Network Theory 
(Law and Hassard 1999). 


Turkle (2006) describes this trajectory in terms of a move in debates on AI from being 
centred around the question of whether machines could ‘really ’ be intelligent, in terms of the 
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capabilities of the objects themselves, to current debates about relational and sociable 
machines (e.g. Breazeal 2002), where the focus is ‘not about the machines ’ capabilities but 
about our own ‘vulnerabilities' . Traditionally, she adds, AI concentrated on building 
engineering systems that impressed by their rationality and cognitive competence—whether 
at playing chess or giving “expert” advice. Relational artefacts, by contrast, are designed to 
impress not so much through their “smarts ” as through their sociability. 


A shadow over this history has been the frequently unfulfilled predictions of the more 
optimistic AI enthusiasts about timescales for the building of intelligent machines that 
resemble humans. This perceived underachievement lay behind the Lighthill (1973) report for 
the UK Science Research Council, which was damning about the achievements and prospects 
of AI. This had a strong negative influence on funding for related research in the 1970s. 
Reservations about what was seen to be continuing AI overhyping were expressed at the e- 
Horizons forum. 


However, Wilks (2006: 6) rejects the notion of AI as a ‘failed project’, as he says ‘it is simply 
everywhere ’. He notes that AI ‘is in the computers on 200-ton planes that land automatically 
in dark and fog and which we trust with our lives; it is in chess programs like IBM’s Big Blue 
that have beaten the world’s champion; and it is in the machine translation programs that 
offer to translate for you any page of an Italian or Japanese newspaper on the web. And AI 
certainly is present in the computer technologies of speech and language’. 


395 



Wilks (ibid) also points out that many AI pioneers had always seen their field’s mission as 
modeling the normal, rather than the ‘superhuman ’, in order to capture “the shorthand of 
reasoning, the tricks that people actually use to cope with everyday life. Only then would we 
understand the machines we have built and trained and avoid them becoming too clever or 
too dangerous. ” It is in these ways that AI has been primarily contributing substantially to 
the development of the kinds of artificial Companions that were the central concern of the 
forum reported here. 


Key social, psychological and ethical issues raised by artificial Companions 

While there was a general acknowledgement that artificial Companions can bring substantial 
practical benefits in a variety of socially valuable applications, contributors emphasized a 
number of deeper issues where these and associated developments raise profound social, 
psychological and ethical issues. 


Rethinking what it means to be human 

A strong theme in workshop discussions revolved around questions about what ACs could 
mean in rethinking notions of human and machine ‘‘intelligence'’ 9 , personhood and the 
implications for relationships between humans and other entities. Sherry Turkle, Director of 
the MIT Initiative on Technology and Self Program, emphasized important implications as 
some societies become increasingly composed of ‘ tethered selves people in continuous 
connection to virtual and real worlds, never alone although often so in terms of direct 
physical contact with other people. In this environment, ACs have a distinctive and important 
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role because they focus on engaged relationships that recognize and attempt to deal with the 
personal emotions and social settings of users and their digital artefacts. 


Such ‘ close encounters ’ with entities that may to some extent become extensions of our 
selves could challenge traditional understanding of the meanings of ‘self and ‘ person hood ’. 
For instance, O’Hara (this volume) argues that much philosophical discussions on memory 
(e.g. Warnock 1987: 1-14) assumes that personhood is more or less a matter of spatio- 
temporal continuity of the body, and that the human mind is more or less identical with the 
human brain. What then of the ability of an AC to build a similar database of memory, which 
could make users dependent on their ability to recall memories shared with their artificial 
Companion? 


Another key focal point of workshop debates was the implications of ACs for ideas about the 
value of authenticity, especially concerning human emotions. Turkle (2006) points out: 
“When we are asked to care for an object, when that object thrives and offers us attention and 
concern, we feel a new level of connection to it.” 


Wilks (2006: 3-4) highlights the implications of this for ACs in the way an elderly Japanese 
woman related to a talking doll called Primo Puel 94 : 

she kissed it while it talked to her, and she said how safe she felt with it there, even 
when chattering away to itself in the next room. It was so much better against 
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loneliness, she said, than praying in front of her dead husband’s shrine, which had 
been no help, she admitted. 


Although the doll is a relatively primitive AC, it can record how Akino moves about the 
house, and can phone the health authorities if sensors in the rooms show that her routine 
changes. Wilks stresses that this illustrates how artificial Companions are already entering 
our society ‘ gently and by ‘ stealth He suggests: “We must think now about their technical 
basis, their limits, what role we want them to have, and how to protect ourselves from them 
and the effects of their arrival, should it become necessary”. 


Figure 2 summarizes some important questions highlighted during the forum. Differing 
perspectives on how to address these are discussed later in the paper. 


Figure 2. Key social, psychological and ethical challenges 

What does it mean to be human when many once seemingly distinctive aspects of being human can 
now be simulated by digital systems? 

What are the limits in computer modelling of human behavior and thinking? 

What do ACs say about the changing relationships between people, and between humans and non¬ 
human entities? Are relationships with non-biological entities intrinsically different? 

How significant is authenticity of feeling in relationships? In particular (see Boden 2007), to what 
extent: 
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1. Could an AC be made to appear to do/feel a certain emotion? 


2. Would the human user believe the AC could do/feel X? 

3. Would we want the user to believe this? 

4. Would this have an effect on the user’s relations with other people? 

Does it matter that ACs can be ‘cheap dates ’ by creating empathy through very simple AI techniques 
(e.g. ‘intelligent’ toys like the Tamagotchi and Paro)? 

Can it be good for us if our experience with relational artefacts is based on a fundamentally deceitful 
interchange in which the artefact persuades us that they know of, and care about, our existence? Or 
might it be good for us in the ‘feel good’ sense, but bad for us in a moral sense? (Turkle 2006). 

Can (or should) empathy or love with an AC be as—or even more—rewarding than with another 
human? 

What does it mean to be an ‘intelligent machine ’ or ‘artificial life ’ (e.g. an artificial Companion)? Do 
these entities have rights and feelings in any meaningful sense? 

What are the implications if ACs and robots develop an ‘alien ’ culture (Winfield 2007) inscrutable to 
humans? 


Balancing potential benefits and harms 

Turkle (2006) articulated a concern frequently expressed during the workshop: that some 
people might find ACs better Companions than humans. She illustrates this by referring to a 
comment she recorded during her research. When talking about her relationship with Aibo, 
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Sony’s household entertainment robot, one woman suggested this device “is better than a real 
dog ... It won’t do dangerous things, and it won’t betray you ... Also, it won’t die suddenly 
and make you feel very sad.’’ This kind of response to a machine was seen as something 
potentially disturbing by some, but welcomed by others who thought such relationship could 
be of great benefit to some people 


The two-edged nature of ACs, as with most ICTs (Dutton 2004), was illustrated frequently 
during the forum. For instance, one of the key advantages of an AC, such as the 
COMPANIONS Senior demonstrator, is that it can acquire intimate knowledge about the 
user’s life. This is essential if it is to gain a rapport with its owner to support his or her needs 
over a long time. However, this also raises potential data security and privacy issues (e.g. if 
the Companion shares that knowledge with other ACs). This issue could be exacerbated if 
ACs and sociable robots develop a shared culture that is quintessentially alien, in effect an 
exo-culture.. .inscrutable to humans, which means that when hots start gossiping with each 
other about you, you will have absolutely no idea what they’re talking about because—unlike 
them—you have no theory of mind for your digital Companions. 


Lowe (this Volume) highlights the benefits artificial Companions can bring by helping their 
users to reframe those choices that can be difficult to make if they require expertise they do 
not have, or when an apparently easy and desirable short-term choice has harder long-term 
consequences (e.g. in relation to a diet or educational requirements). Here, AC support could 
be helpful as an ‘enforcer ’ (e.g. to ensure the full course of antibiotics is taken) or conscience 
(e.g. on what not to eat when on a particular diet). He graphically explained this process as: 
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“When the sirens of plan-breaking temptation sing, it may be your Companion that ties you to 
the mast”. 


The user could face severe harm if an AC in an ‘ enforcer ’ role is imposed by a state, 
organization or individual with malign intent. However, Lowe also argues that by 
persistently prioritizing the long-term over the immediate and short-term or by relying on 
information about the user and her situation that is over-rigid or outdated even an otherwise 
benign AC may reduce the spontaneity and quality of its owner’s life. 


Underlying clashes of values and perceptions 

Some forum participants welcomed the possibility of developing artefacts that could be of 
great benefit to people who, for various reasons, are unable to have equivalently fulfilling 
relationships with people. For instance, Levy (This volume) states he is 

convinced that, within twenty years at the latest, there will be artificial emotion 
technologies that can not only simulate a full range of human emotions and their 
appropriate responses, but also exhibit non-human emotions that are peculiar to 
computers (and robots). 

At the workshop, Margaret Boden argued that it would be “ tragic ” to mistake falling in love 
with a robot or AC with falling in love in the “deepest most important sense of personal 
human love” (e.g. see Fisher 1990). However, others welcomed, or at least accepted, the 
validity of love with an AC as one type of love across a spectrum ranging from human love, 
through pets and plants to emotional attachments to inanimate object like mobile phones. 
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Levy (This volume), for instance, welcomes of the prospect of people “not only be falling in 
love with software Companions, but also having sex with them”. 


One discussion related to the existence of possible underlying ‘ essentialisms ’ influencing 
these conflicting perceptions about the degree to which people welcomed or were 
apprehensive about AC developments. As indicated in Figure 3, one such extreme could be 
characterized as being that of ‘humanists’ who privilege human emotional authenticity and 
the uniqueness of human intelligence. At the opposite pole could be the view of the more 
extreme proponents of artificial life who would like to create a new, more perfect set of 
‘ beings' (e.g. see Helmreich 1997). A spectrum of views between these poles was reflected at 
the forum. 


Figure 3. Conflicting perspectives: Humanist v Artificial Lifer 


'Artificial Lifer ’ 

‘Humanist ’ 
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Human relationships are the most important 

Human and other biological life forms are unique 
and have a unique value 

Authenticity of human understanding and 
emotions is essential to engaged relationships 

ACs should primarily seek to mediate and 
support human-human communication 

ACs should support human needs, including 
being a servant or even ‘slave’ 

Computer modelling of human emotions and 
behavior is too difficult to achieve in a realistic 
manner 

Users should make influential contributions to the 
design of AC capabilities 

Machine intelligence is fundamentally different 
to human intelligence 

There can be no meaningful, reciprocated loving 
relationship with artificial life. 


Relationships with ACs can be as—or more— 
rewarding than human ones 

Artificial life artefacts can have equivalent value 
and rights to other life forms 

‘Fit for a purpose’ functionality is a good enough 
aim for AC and intelligent robot developments 

Interactions between a human and AC can be a 
valuable end in itself. 

ACs represents a new breed of autonomous 
artificial life of value in its own right. 

Within a reasonable period it will be feasible to 
simulate a full range of human emotions and their 
appropriate responses. 

The focus should be on technological innovation 
to create new choices for users. 

Machine intelligence can simulate and even 
emulate human intelligence 

Falling in love with a AC or robot is as natural as 
other forms of love. 


Challenges to creating effective appropriate artificial Companions 
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Wilks (This volume) says there is still some confusion and lack of consensus among 
specialists in this field over what is desirable in the self-presentation of a possible long-term 
artificial Companion, how we should view or deal with them and what social and personal 
roles we want them to adopt with us. This is reinforced by Sloman’s (This volume) comment 
that the detailed requirements for ACs are “not at all obvious, and will be found to have 
implications that make the design task very difficult in ways that have not been noticed”. 
However, he adds that overcoming these difficulties is 1 perhaps not impossible if we analyse 
the problems properly ’. 


Much time at the workshop was devoted to discussion of various approaches to designing and 
building artificial Companions which would fulfill the main hopes in this field. This included 
an acknowledgement of the dangers of basing ACs on computing models that eliminate 
essential features of the real world being simulated to make the model manageable—without 
taking account of the richness, subtleties and complexities of human motivation and behavior, 
social contexts and the complex interactions that take place with the physical environments in 
which we (and perhaps AC robots) move and live. Figure 4 summarizes some key questions 
raised about designing and building artificial Companions. 


Figure 4. The main artificial Companion design and build questions 


• What are the most appropriate Human-AC relationships? 

• To what extent should ACs complement or replace human-human communication? 
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How human-like should the AC be? 


• What are the limits of computational modelling of personal emotions, intelligence and behavior 
within realistic social and physical environments? 

• How best can potential users of ACs influence designs, particularly when they may be unaware of 
what can be achieved and how their own requirements may change once they experience life with 
an artificial Companion? 

• How can the user requirements of vulnerable groups be elicited most effectively, such as for the 
elderly (see Newell 2007). 

• What are the technical and cost constraints determining the feasibility of building ACs with 
appropriate performance levels (e.g. to achieve believable interactions, such as the realism of 
graphics in well-financed movie animation)? 

• How can ACs be made as flexible as possible to enable them to meet, in a cost-effective manner, 
the distinctive needs of each user in their particular situations—and their evolving requirements 
over a long period of time? 

• How transparent should an AC be in making clear the non-human nature of emotions and its 
machine-like ‘thinking’ processes? 

• Should ACs be governed by ethical ‘rules’ relating to their behavior towards users? 

• Should ACs be regarded as having ‘rights’ (e.g. against ‘abuse’)? 


The next section explores potential solutions and some barriers to resolving the design 
questions outlined here. This is followed by a discussion on the main policy issues relating to 
the growing use of ACs. These need to be better understood if a framework is to be created 


405 



that supports wider use of artificial Companions, while minimizing potential hanns. The 
paper concludes with a summary of the longer-term implications of living in societies 
populated by a growing range of artificial Companions in continuous close engagements with 
people. 


Approaches to designing and building artificial Companions 


What features are likely to make a ‘good Companion ’? 

As indicated in Figure 1 above, there are a wide range of entities that could be regarded as 
artificial Companions. However, a number of broad AC types and core features emerged 
from discussions at the e-Horizons forum. These were broadly captured in Wilks’ (This 
volume) description of the qualities of a typical Victorian lady’s Companion. Many of these, 
he noted, would now be considered as part of the general caring and social services. Floridi 
(This volume) identifies three classifications of such Companions: as social workers; service 
providers in specific application contexts such as education, health, safety and 
communication; and memory keepers, as in the Memories for Life project (see Appendix 1). 
Boden (This volume) expresses concern about the privacy aspects of being a “ confidant ’ 
gleaning intimate personal information, and so prefers an emphasis on ACs as 
“conversationalists'". 


Figure 5 summarizes some features the designers of ACs could consider, based on the 
Victorian lady’s Companion and other characteristics suggested at the forum. Many of these 
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characteristics are actually extremely difficult even for humans to achieve (e.g. being 
supportive, engaging in long-term relationships, correctly interpreting emotions, being witty). 


Figure 5. Potentially desirable features of an artificial Companion 

1. Able to recognize its ‘owned as an individual distinguishable from other people, animals and things 
in the environment. 

2. Able to understand its owner’s emotions and intentions, and respond in an appropriate manner. 

3. Self-presentation by the AC that is coherent and believable. 

4. Dependable, predictable behavior in the service of the owner 

5. Well-informed, particularly about its owner’s needs, memories and social relationships 

6. Supportive as a mediator in human-human communication, to complement existing communication 
methods (e.g. as a more friendly intelligent interface to the Web). 

7. Long-term relationship if possible 

8. Able to act as a caring expert advisor. 

9. Independent, not requiring much effort from the user to use, sustain and nurture. 

10. A discreet confidant in knowing what should or should not be communicated, and who or what 
(e.g. another AC) can be told particular information 

11. Knowing its place relative to the human it supports and serves 
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12. Trustworthy 


13. Responses firmly under control to best suit its owner 

14. Good conversationalist and sympathetic listener 

15. Polite where necessary, but firm if that is in the owner’s interest 

16. Modest and realistic about its own capabilities 

17. Diverting, able to be witty and entertaining where appropriate 

18. Cheerful even when faced with difficult problems 

19. Specific looks not important provided they do not antagonize or irritate 

20. Operationally reliable, resilient and requiring little special maintenance 


One of the most distinctive features of ACs as digital artefacts is their integral need to engage 
with the emotional states of their users. This is emphasized by Cowie (This Volume), who 
points to the ‘pervasive emotions ’ that colour most of life as being of particular importance to 
ACs—for which briefer intense emotional episodes may be less significance. Human- 
computer interfaces, he commented, were traditionally oriented towards people who could be 
expected, at least temporarily, to adopt a more or less unemotional stance to the task in hand. 
“The populations who need artificial Companions are likely to find it harder than average to 
adopt and sustain unemotional stances,” he adds. 

Cowie (ibid) sounds a note of caution to designers of ACs: “Among the commonest 
misunderstandings is the idea that engaging with emotion is necessarily about devices that 
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mimic human emotional life.” Nevertheless, he emphasizes that a basic level of attention to 
emotion cannot be left out of work in this field. 


How human should an artificial Companion attempt to be? 


The question of how much like a human the AC should present itself is central to debates 
about the design of an AC’s self-presentation and behavior. There was a general feeling at the 
forum that looking human was not an intrinsic requirement (e.g. see Wilks 2006; Pulman 
This volume), provided the form of presentation is appropriate to the applications and is 
believable (Romano, This volume). 


Different views were expressed about the relevance of the concept of an ‘uncanny valley', 
which is said to affect the believability of synthetic human-like presentations. Romano 
explains that this valley was initially identified by Mori (1970) to represent the way 
emotional responses to an artificially generated character is thought to move through a curve 
that rises slowly up to a point, then descends rapidly into the uncanny valley as the 
presentation increasingly resembles a human—but not perfectly. After the lowest point is 
reached, the curve rises almost vertically to induce a positive reaction in humans to the 
presence of actual humans, which doesn’t occur in relation to artificial entities that may 
closely resemble humans. 
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Cowie questions the uncanny valley thesis, as he regards it as a fallacy to suggest there is a 
fundamental obstacle to the acceptance of devices that are human-like. Rather, he sees “signs 
of a long valley lying in wait for any approach to human-like qualities, or isolated potholes 
that a sensible team will steer round”. Nevertheless, he acknowledges that “o near-miss can 
be a disaster” in tenns of getting close to a human-like appearance. Predictability of 
behavior, with expectations tailored to what can be delivered, is seen by many to be more 
important than human-like features in many AC presentations. Figure 6 summarizes some of 
the specific advice on designing an AC highlighted at the workshop. 


Figure 6. Examples of how to design effective AC user interactions 


1. Ensure there is coherence in the appearance, characterization and behavior of an AC. For 
instance, having photorealistic people does not make them necessarily more believable if their 
behavior, speech and language are not of the same level as the visual qualities. Any discordant 
element can make the character unbelievable. 

2. Consider the aesthetic qualities of the AC’s movements, its behavior and its ability to interpret 
and respond verbally and non-verbally to the human user (e.g. humour and politeness can make a 
character more human-like). 

3. Build-in an autonomous modelling capability to enable the AC to generate its own behavior 
according to its perception of the interaction with the user and the environment, generating 
appropriate expressions, emotions and credible behavior. 

4. Understand how to recognize and respond to attentive and motivational cues in real-time, in a 
manner that achieves natural dialogue behavior and adequate response times. This requires 
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recognition and response processes to be very fast, and synchronized, in order not to be perceived 


as awkward. 

5. Distinguish decreasing engagement from those grounding problems where the speaker and 
listener have conflicting views regarding which object the conversation is about. For example, 
the fact that a user looks at an object that is not meant by the speaker does not necessarily mean 
the user is no longer engaged in a conversation. 

6. Find the right level of sensitivity in responding to the user’s attentive and motivational state, for 
instance by not overreacting to the user’s state (e.g. Eichner et al. 2007). 

7. The AC need not always display its emotion immediately in response to a user cue, as in some 
circumstance it may be more appropriate to mask, disguise or delay the response. 

Understanding and meeting actual user requirements 

A prime motivation for the development of HCI as a discipline in the 1960s and 70s was a 
growing awareness that most computer systems had been designed by technical experts, with 
little conception of what users actually wanted and needed. Despite many HCI advances and 
a greater awareness of the importance of meeting user requirements, many forum participants 
expressed concern that AI innovations like artificial Companions are still driven too much by 
the perceptions and wishes of technically-oriented designers. 


Prioritizing user-led design requirements 

The frequently very close personal engagements envisaged for human-AC relations, 
involving sensitive feelings and intimate infonnation, led many at the forum to emphasize the 
need for a creative and flexible understanding of human relations and psychology to be the 
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foundation of artificial Companion design and development. Newell strongly argues that it is 
important to understand the characteristics of potential users of artificial Companions, as they 
can be very different from the characteristics generally favoured by designers of new 
technologies and their traditional user bases. 


This is particularly significant for those users in the vulnerable groups who are likely have a 
special need for such Companions. For example, Newell (ibid) explains: 

Older people have multiple minor disabilities, sensory, motor, and cognitive, and may 
also have major disabilities. The vast majority will have substantially different 
experiences of, and emotional attitudes to, new technologies than younger cohorts. 
Many of them will not understand, or be familiar with the jargon, metaphors, or 
methods of operation of new technologies, and have major lack of confidence in their 
ability to use them. 


Successful ACs that meet the needs of such groups require the development and adoption of 
appropriate methods of eliciting knowledge of their needs. Some potential users with no 
experience of associated technologies may not even be aware of the feasibility of technical 
capabilities that could provide the support they would like. Any requirements elicitation 
methods used for such potential AC users should take account of their special physical, 
sensory and cognitive characteristics, together with their experiences of, and attitudes 
towards, current technologies. Novel requirements-gathering techniques are likely to be 
needed for specific groups. For instance, Newell (ibid) says theatre techniques that dramatize 
future possibilities 95 have proved to be effective with older people. 


412 



It will also be important to determine what safeguards are required, for example to protect 
intimate AC-held information that the system’s owners might not want to be divulged to 
close relatives. In addition, attempts should be made to find out why some people may decide 
they do not want to use certain types of technology in particular situations. Such ‘ digital 
choices ’ are evident, for example, in decisions by elderly people not to use the Internet even 
when it is freely available for other members of the family in their home (Dutton 2005). 
Turkle described how a disabled colleague had been abused by nurses in a care home. 
Although he is aware of the development of ‘ nursebots ’ to care for people in such homes, he 
told her that he would still prefer to be dealt with by a human nurse, even one of his abusers, 
as he could still connect with the ‘ human narrative ’ of that person. 


Allowing adjustments to meet bespoke user needs 

In constructing artificial Companions, the potential to be adapted to different user needs 
should be considered as a fundamental capability. This would help to support people to tailor 
their Companion and the way they use it to fit varying configurations of their own 
relationships with people, pets, technologies and other entities. It would also allow the AC to 
evolve with changing user requirements. Finding out about, and better understanding, such 
diversity in user requirements will be assisted by a user-led approach to AC design. 


A core system for an AC could therefore be designed to meet the common needs of a 
substantial sector of the user base, but allow adjustability by user selection and controls (e.g. 
by including a software library of user-chosen options). For instance, the kind of digital 
learning Companion proposed by Davies and Eynon (This volume)—which seeks to support 
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the needs of adult learners who have difficulties in other environments—could be adapted to 


lit different learning styles (e.g. Pask 1976). 96 


Policy to help shape appropriate artificial Companion uses 


The national and international policy significance of artificial Companions and related AI 
developments is indicated by the support given by the Japanese government and industry to 
the development and use of such innovations to support its ageing population, thereby 
reducing the potential dependence on immigrant labour. The opportunities and concerns 
outlined in this paper also raise other significant policy issues. 


The targeting of artificial Companions at supporting vulnerable groups in society could 
clearly help to close social discriminations and digital divides. However, this could also lead 
to such gaps widening if policy measures are not taken to ensure groups without the financial 
resources or skills and knowhow to make effective use of these systems are not given 
appropriate assistance. 


Legal aspects, such as liability (e.g. see Wilks and Ballim 1990) were frequently mentioned. 
Wilks (This volume) emphasizes the practical significance of questions about where 
responsibility and blame may lie when an AC acts as a person’s agent and something goes 
wrong. He explains: 
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At the moment, Anglo-American law has no real notion of any responsible entity 
except a human, if we exclude Acts of God in insurance policies. The only possible 
exception here is dogs, which occupy a special place in English law, at least, and 
seem to have certain rights and attributions of character separate from their owners. 
If one keeps a tiger, one is totally responsible for whatever damage it does, because it 
is ferae naturae, a wild beast. Dogs, however, seem to occupy a middle ground as 
responsible agents, and an owner may not be responsible unless the dog is known to 
be of “bad character. 


Wilks (ibid: 4) wonders whether AC developments could influence changes in the law so that 
things that are not human could be liable for damages. Until now, if a machine goes wrong it 
is always the maker or the programmer, or their company, which is at fault. Bryson (2000) 
argues that liability should always be firmly associated with a human, although whether it is 
the owner/operator or designer who is to be liable would remain an issue to resolve. A 
wholly new possibility, which would meet most needs, would be that each Companion 
becomes a limited company: Thus, at a stroke, giving it an identity and its 
creators/stakeholders/directors protection from damage suits. 


Turkle expressed a fear that many at the leading edge of digital developments have begun to 
become too complacent about the potential for ICTs to be used to heighten surveillance and 
control over citizens. She said she has noticed a trend for many digital technology enthusiasts 
to argue along the lines: 
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All information is good. We are observed all the time. If we give up information now 
freely, say on social networks like FaceBook and MySpace, we will be less troubled 
later. So there is no need to worry about surveillance. And in any case if you have 
nothing to hide there is no need to be concerned. 

However, a great deal of such information has been used to repress people and political 
movements, so it seems over-confident to imagine that no regime would ever misuse data 
within your, or your data’s, lifetime. 


This view could reduce pressure to establish and implement clear policy guidelines to protect 
citizens’ privacy and other rights which could be infringed by increased digital surveillance 
and privacy intrusions. It is possible that the ability of ACs to gather intimate personal 
information for what are seen to be socially valuable purpose could add a new dimension to 
this kind of abuse. The way even primitive ''intelligent'’ artefacts can be, in Turkle’s phrase, 

‘cheap dates ’ in convincing people of their emotional sincerity also creates new opportunities 
for those with harmful intent to commit deception and fraud. 


Figure 7. Key policy issues on the creation and use of artificial Companions 

Policy area Key issues 

Digital divides Support to ensure appropriate ACs are developed and made accessible to all 

individuals in vulnerable groups, such as the elderly, disabled, ill and learners with 
special needs 
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Capacity and skill building to help vulnerable groups make informed decisions 


Legal 


Privacy and 
surveillance 


about which ACs they could choose to use (or not use) 

Divides between older generations and those who are growing up in an always-on 
digital world 

The degree to which plans to provide ACs to vulnerable groups may reduce 
pressure to look for other, perhaps better, solutions for their care 

Liability for incorrect advice or actions by an AC; possibly linked to each 
Companion being a registered company. 

Implications of giving ‘informed consent’ for an AC to act on behalf of a person 
(e.g. after death) 

Status of ‘professional privilege’ when an AC gains confidential information in a 
particular ‘expert’ role (e.g. as medical advisor) 

Control over what biographical information is recorded by an AC; its protection 
from unwarranted access and change; its use after a person’s death; and its 
eventual possible deletion. 

Ownership of the biographical narrative acquired by an AC. 

Rights of others (e.g. spouse, children, doctor, police, government, business 
partner) to access biographical information gathered by an AC. 

Use of ACs as a business strategy to acquire personal information from customers 
(e.g. if a supermarket were to give simple ACs for free to chat about shopping with 
lonely and bored customers). 
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Sharing of information by an AC with other ACs. 


Safety Deliberate deception to gain a person’s confidence, similar to spam email 

‘phishing’ for bank details, capable of being used for economic and emotional 
exploitation of the user. 

Identity theft (even after death) using information provided confidentially by an 
AC’s owner. 

Socio-economic Using ACs to meet social, economic and political requirements, such as caring for 
the elderly or reducing the need for immigrant labour. 


The future: Can we live in harmony with artificial Companions? 

The boundary-spanning value of artificial Companions 

The wide range of issues outlined above indicate that the artificial Companion concept 
provides a very useful tool or, speculum mentis, spanning many disciplines, for examining 
issues at the boundary where the human meets the artificial and computational. For instance, 
it challenges computer scientists to consider diverse issues related to notions like that of an 
AC being an extension of the user’s "self. They therefore need to consider deeply in their 
design processes the relevant non-technical factors determining the outcome of their 
innovations. Likewise, it forces social scientists to think of design issues for the computer 
sciences, such as what a Companion should look like—and what kinds of communication 
will be essential or sufficient, both between Companions and in the kinds of social relations 
between humans and ACs that will tend to be fostered or inhibited. 
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One of the main future design challenges is to better understand and deal with the real world 
environment in which the user of a virtual artificial Companion lives, or in which an 
embodied AC robot moves. Sloman (2007) claims that giving machines an understanding of 
the physical and geometrical shapes, processes and causal interactions that occur in an 
ordinary house “is currently far beyond the state of the art” as “it has proved extremely 
difficult to give machines the kind of intuitive understanding required for creative problem¬ 
solving in novel physical situations ” 97 . 


Future implications beyond AC developments, as such, were highlighted by Turkle’s concern 
about the implication for notions of authenticity as more people become ‘ tethered selves ’, 
always intertwined within virtual and real worlds (see also Reeves and Nass 1996). This 
places forum discussions at the centre of broader work on the social implication of the 
Internet and other ICTs 98 . At the same time, the ‘cyber intimacies'’ displayed in some 
people’s reaction to relatively simple ACs indicate that these developments could be opening 
distinctive new challenges concerning the relationships between people and machines. 


For instance, from her research Turkle (2006) reports the reaction of a woman sitting with the 
robot Paro, a seal-like device advertised as the first ‘therapeutic robot’ because of its 
ostensibly positive effects on the ill, the elderly and the emotionally troubled. Depressed 
because her son had abandoned her, the woman turned to Paro and, as she stroked it, she said: 
“Yes, you’re sad, aren’t you. It’s tough out there. Yes, it’s hard.” Turkle observes: “And then 
she pets the robot once again, attempting to provide it with comfort. And in so doing, she 
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tries to comfort herself’. Levy (this volume), along with other contributors, refers to the 
strength of affection and caring shown towards Tamagotchi artificial pets, which he says can 
be as strong as the affection shown for live pets. 


At the start of the workshop, Bill Dutton, Director of the Oil and Co-Director of the e- 
Horizon’s Institute, showed a video of Apple Computer’s ‘Knowledge Navigator 
Conceived around 1987, this used actors to show what a software-based online desk-top 
personal assistant could eventually become: a simulated human prompting, advising and 
responding to a manager in the way humans would naturally communicate in an office. To 
some, this could have reinforced the view that AI has once again underachieved in failing to 
turn vision into reality. For others, it remains a goal researchers are getting ever closer to 
achieving. 


However, it would be wrong to dismiss such prospects and their implications as belonging to 
the realm of science fantasy. The actual experiences of 'cyberintimacies'’ explored at the 
workshop, as well as the advances that continue to be made to make ACs more believable—if 
not yet at Knowledge Navigator perfection—indicate that the related issues raised in this 
paper will need to be addressed in many practical ways in the coming years. 


Wilks drew on the earliest science fiction vision of an artificial Companion, in Mary 
Shelley’s Frankenstein, to warn that we must “take seriously the possibility that everything 
may turn out differently from what we expect and Companions, however effective, may be 
less loved and less loveable than we might wish”. This kind of dystopian fear, as well as the 
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many Utopian promises of caring artificial Companions, lay behind a core thread running 
through the forum. This was a search for an answer to the question: ‘ What are the 
appropriate relationships between people and artificial Companions?' 


Can a person engage in an I-Thou relationship with an artificial Companion? 

Answers to the question about what makes an appropriate artificial Companion can challenge 
traditional ideas of authenticity and selfhood, for example by claims that it is leading to 
emergent new meanings of intelligence and emotions in ACs. These were crystallized at the 
forum around a discussion of whether human-computer relation can ever have the quality of 
the ‘I-Thou’ ethical relationship, as described by the philosopher Martin Buber (2004). 


What is it to be human? 

In her talk, Turkle reported that she had found many digital technology and AI enthusiasts 
have begun talking about people having an I-Thou relationships with cybersystems. She feels 
this profoundly misunderstands Buber’s conception of such a relationship. For this, she says, 
there needs to be “mutual understanding of deep life expectancies” and the biological 
lifecycle. But she believes this cannot happen “if you think someone’s there but there isn’t”. 
From this perspective, it is insufficient for an artefact without genuine emotions to seem to 
offer sympathy (e.g. an AC acting as a psychiatric counsellor talking to a person about sibling 
rivalry when it never had a mother). 
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Dutton suggested we may be asking too much of ACs in suggesting they can become 
‘human-like ’ in their understanding of the emotional states of others, when people find those 
judgements so difficult to make accurately. On the other hand, there has been much evidence 
traditionally that human relationships can be effective even without any special or correct 
intentions on the side of one participant. For instance, a psychiatrist may not always be 
listening intently to a patient, who may still draw benefit from that therapeutic session 100 ; and 
a confession to a Catholic priest could bring comfort, even if the priest was thinking of 
something else, but provided a seemingly appropriate response. 


Nevertheless, as discussed above, developments like artificial Companions are challenging 
traditional notions of the self, with computer systems in some cases becoming virtual 
extensions of a tethered self. Lowe pointed to the complexities of the relationship between 
the ‘self of a user and the ‘other ’ of an artificial Companion. He said that ACs ‘will need to 
be distinct and independent enough for us to treat their advice, information, and 
representation of our rules as real constraints on our action, but sufficiently closely aligned 
and sensitive to our goals that we do not feel them as an imposition’. 


O’Hara sees the human-AC relationship closer to the Johnson-Boswell, biographer-subject, 
friendship. In this, the AC acquires its owner’s biographical knowledge from conversations 
with the person and other relevant sources, building up a perception of the subject’s life, but 
not acquiring the life itself. He suggests an insight into the emerging relationship with ACs 
can be gleaned from a comment by Zizek (1997: 137) on how people relate to the avatars 
they have created to represent themselves in a virtual world: 
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on the one hand we maintain an attitude of external distance, ofplaying with false 
images. ... On the other hand, the screen persona I create for myself can be 'more 
myself than my 'real-life ’persona ... in so far as it reveals aspects of myself I would 
never dare to admit in real life. 

Wilks speculates that future artificial Companions could act as an owner’s agent: e.g. on the 
Internet or, further in the future, perhaps holding power of attorney in case of an owner’s 
incapacity or, with the owner’s advance permission, of being a source of conversational 
comfort for relatives after the owner’s death. O’Hara (2007) foresees the possibility of an 
artificial Companion functioning on the owner’s behalf after the bodily death of that person 
(e.g. for overseeing and administering trust funds and the execution of wills). 


What kinds of ethics should govern human-AC relations 

Science fiction writer Isaac Asimov’s (1993) three laws of robotics was an early attempt to 
create an ethical code to govern the relationships between people and intelligent machines 101 : 

1. A robot may not injure a human being or, through inaction, allow a human being to 
come to harm. 

2. A robot must obey the orders given it by human beings except where such orders 
would conflict with the First Law. 

3. A robot must protect its own existence as long as such protection does not conflict 
with the First or Second Law. 

One activity characteristic of artificial Companions where such ethical guidance is likely to 
be of value is in controlling the sharing of sensitive information. This goes beyond preventing 
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unwarranted access to questions like to whom (or what other AC) certain infonnation can be 
disclosed, and in what circumstances. 


The ethical stance of humans towards artificial Companions was also highlighted at the 
forum. For instance, Bryson recalls: “I was astonished during my own experience of working 
on (non-functional) humanoid robots in the mid 1990s by how many well-educated 
colleagues asserted immediately (without prompting) that unplugging such a robot would be 
unethical.” Bryson and Kime (1998) have argued that such deep concern with (or fear of) the 
well-being of AI artifacts results from a misattribution of human identity and therefore our 
empathy. Bryson’s suggestion that the best way of viewing ACs or robots is as slaves because 
they are “wholly owned and designed by us, we determine their goals and desires” opened a 
discussion on broader ethics of ‘abusing' artificial artefacts, and whether a tenn like ‘ abuse ’ 
has a real meaning in this context. 


Cowie identified three main issues in the ethics of artificial Companions’ engagement with 
emotion: 

1. Deception. What is the ethical status of building a device whose behavior signals 
emotions that it does not actually feel, in any straightforward sense? Wilks (2006: 6) 
believes ‘Companions are not at all about fooling us as to their true natures’, as in the 
case of an updated version of the ‘Turing test’ scenario where (an originally teletype- 
based) computer passed the test if it could make the user think a person was hidden at 
the other end of the line. 
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2. The ‘lotus eater’ problem. What if making a Companion too emotionally engaging 
risks eroding the person’s motivation for engaging with human beings, who are not 
always emotionally engaging. This could increase isolation instead of reducing it. 

3. Technical limits of emotion detection. For the foreseeable future, it is likely that 
artificial Companions will have a relatively modest ability to recognize emotion- 
related attributes. This could lead to misattributions that could seriously affect what 
happens to the person (e.g. if a false attribution of a negative mood in the AC’s owner 
is fed by the Companion into the medical or care system, such as by the AC calling— 
or failing to call—an emergency service in the wrong circumstance). 


Meeting future real and virtual needs 


As always in discussions on how to develop advanced innovations for an unknowable long¬ 
term future, the most frequent design advice on artificial Companions was to make them as 
adjustable as possible to changing needs. Fulfilling Wilks’ goal of providing a more friendly 
interface to the Internet to help manage its oceans of overflowing infonnation would in itself 
be a significant contribution to giving people more control over ICTs. The more naturally 
human the dialogue between ACs and their users become, the more smoothly will ACs be 
able to adapt to the changing requirements of their owners over a long period, without the 
user having to continually learn new skills and knowledge—and forget old ones—in order to 
interact effectively with the artificial Companion. 
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New forms of artificial life are also likely to emerge from bioengineering developments, 
including the introduction of biological innovations into traditional AI robotics 102 . A critical 
divide for the future could be the generational one, particularly in terms of issues like 
authenticity, as younger generations may find interacting with artificial beings more ‘natural’ 
and appealing than earlier generations. For instance, Turkle describes how the reaction of 
some young people to a real tortoise at a Darwin exhibition in New York suggests that, in 
certain circumstances, the emerging digital-savvy generation will think that “aliveness 
doesn’t seem worth the trouble”. She reports a 12-year-old girl adamantly stating: “For what 
the turtles do, you didn’t have to have the live ones.” Her father looked at her, 
uncomprehending: “But the point is that they are real, that’s the whole point.” 


Appendix 1. Resources 

The following resources can help to explore in more depth the issue discussed in this paper. 
They are related to material provided and references at the e-Horizons workshop on artificial 
Companions. 


Examples of artificial Companions 

Aibo. Sony’s ‘intelligent’ robot dog Companion/pet (http://support.sony- 
europe.com/aibo/index.asp). 
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BASIC (Believable Adaptable Socially Intelligent Character). Research project at Sheffield 
University Department of Computer Science developing synthetically generated characters to 
produce a believable graphical emotional response to a direct interaction of the user, another 
character in the BASIC world, or the emotionally charged atmosphere of the environment 
(http://www.dcs.shef.ac.uk/~daniela/gallery.html). 

Beating the Blues. An interactive, computer-based therapy system developed at King’s 
College London to assist people suffering from depression and/or anxiety. Based on 
Cognitive Behavioral Therapy (http://wwww.mentalhealthcare.org.uk/content/?id=98). 

Ecobot. An energetically autonomous robot built at the Bristol Robotics Laboratory using 
Microbial Fuel Cell technology to extract electrical energy from refined foods such as sugar 
and unrefined foods such as insects and fruit (http://www.brl.ac.uk/projects/index.html). 

ESP (Emotional-Social Intelligence Prosthesis Technology). Developed by MIT Media Lab’s 
Affective Computing Group as a wearable system that can augment and enhance the user’s 
ability to sense non-verbal cues (e.g. facial expressions and tone of voice). It uses common- 
sense knowledge about people that may not be natural for certain groups, such as those 
diagnosed with autism, (http://affect.media.mit.edu/projects.php?id=1935). 

Furby. An ‘intelligent’ toy that looks like a furry owl. At first it speaks only its unique 
Furbish language, but it is programmed to speak less Furbish when it leams more English as 
it is used more. Voice recognition and increasingly complex facial movements are among the 
enhancements being introduced 

(http://www.hasbro.com/monkeybartv/default.cfm?page=SearchResults&criteria=furby). 

Greta. 3D virtual agent, part of the HUMAINE project (Pelachaud 2007 and http://emotion- 
research.net/deliverables/D6f_vl.5%20Feb%2007.pdf). 
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KASPAR. A child-sized humanoid robot developed by the Adaptive Systems Research 
Group at the University of Hertfordshire to study human-robot interaction as part of the 
RobotCub Project. It can be used for developmental studies, including as therapeutic or 
educational tools to encourage social interaction skills in children with autism 
(http://kaspar.feis.herts.ac.uk). 

Kismet. Developed by the Sociable Machines Project in MIT’s Humanoid Robotics Group. 
Able to engage people in natural and expressive face-to-face interaction 
(http://www.ai.mit.edu/projects/humanoid-robotics-group/kismet/kismet.html). 

Laura. A virtual fitness health adviser that can ask the user questions and respond 
empathetically and encouragingly to the answers (see Bickmore 2003 and 
http://www.ccs.neu.edu/home/bickmore/agents/). 

Nabaztag. A programmable Wi-Fi wireless-enabled talking device that looks like a rabbit. It 
has with movable ears and can speak information it downloads from the Internet, such as 
weather forecasts and email alerts (http://www.nabaztag.com). 

Nursebot Pearl (Personal Robotic Assistants for the Elderly). An interdisciplinary university 
research initiative at the University of Pittsburgh and Carnegie Mellon University focused on 
robotic technology for the elderly, particularly mobile personal service robots 
(http://www.cs.cmu.edu/~nursebot/). 

Paro. Developed by the Japanese National Institute of Advanced Industrial Science and 
Technology as a ‘mental commitment robot’. It is designed to interact with human beings to 
make them feel emotional attachment to its embodiment as a cuddly baby seal. 
(http://paro.jp/english/index.html). 
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Primo Puel. A talking doll originally designed to be a substitute boyfriend for young single 
girls in the Japanese workforce, but has become unexpectedly popular among elderly people 
across Japan; has a vocabulary of a few hundred words and can talk, laugh and even ask for a 
kiss (BBC News 24 2007). 

Radar (Reflective Agents with Distributed Adaptive Reasoning). Developments at Carnegie 
Mellon University’s Robotics Institute focused on creating a cognitive assistant that 
embodies machine learning technology that is able to function ‘in the wild’—by using 
technology that need not be tuned by experts, and for which the person using it need not be 
trained in any special way (http://www.radar.cs.cmu.edu/). 

Roomba. Robot vacuum cleaner from iRobot Corp, whose seemingly autonomous 
movements can make it seem ‘alive’, at least to real pets (http://www.irobot.com). 

RoboCup. An international challenge aiming to develop a team of fully autonomous 
humanoid robots that can win against the human world champion team in soccer—by the 
year 2050 (http://www.robocup.org). 

Tamagotchi. A virtual creature that ‘lives’ on small screens housed in a small plastic egg. It 
can communicate its needs and ‘grow’ from a ‘child’ to ‘healthy adult’ if looked after, but 
dies if its needs are not met (http://www.tamagotchi.com). 

Virtual Woman. Software that presents a 3D animated image of a woman who can engage the 
user in conversation. Users can choose characteristics of the presentation—such a ethnic type, 
personality and clothing—to personalize the image of their own Virtual Woman 
(http://virtualwoman.net). 
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Related research projects 


Adaptive Systems Research Group, University of Hertfordshire. A multidisciplinary group 
studying artificial life, socially intelligent agents and AI (http://adapsys.feis.herts.ac.uk). 

Affective Computing Research Group, MIT Media Laboratory. Founded and directed by 
Rosalind Picard, the group focuses on computing developments that relate to, arise from or 
deliberately influence emotion or other affective phenomena 
(http://affect.media.mit.edu/index.php). 

AMID A (Augmented Multi-party Interaction with Distance Access). European Commission 
project addressing many scientific challenges in multimodal processing, focusing on live 
meetings with remote participants 

(http://www.onderzoekinformatie.nl/en/oi/nod/onderzoek/OND1320459/). 

University of Bath. The AmonI (Artificial Models of Natural Intelligence) group is seeking to 
understand human and animal intelligence by engaging in research in the natural sciences, 
while using their knowledge as engineers to develop software tools and AI techniques to 
make their models more accurate and easier to develop 

(http://www.cs.bath.ac.uk/ai/AmonI.html). The University’s Media Technology Research 
Centre studies computer technology for animation, graphics, image processing, music, 
rendering and virtual reality, including the Animating Virtual Humans project that has 
developed a Dynamic Emotion Representation (DER) for use in controlling the 
expressiveness of virtual characters (http://www.cs.bath.ac.uk/~pjw/media/avatars.htm). 

Bristol Robotics Laboratory. A multidisciplinary venture from the University of Bristol and 
the University of the West of England. It includes work on the Ecobot and a range of 
developments in the Humanoid Robotics and Social Interaction group. The Laboratory leads 
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the EPSRC UK public engagement network ‘Walking with Robots’, one aim of which is to 
create public debate about the ethical questions raised by intelligent robots in society 
(http://www.brl.ac.uk/projects/index.html). 

CALLAS (Conveying Affectiveness in Leading-Edge Living Adaptive Systems). European 
Commission project designing and developing a multimodal architecture that includes 
emotional aspects (http://www.callas-newmedia.eu). 

CALO (Cognitive Assistant that Leams and Organizes). US project led by SRI 
International’s Artificial Intelligence Centre. Supports decision making by creating cognitive 
software systems that can reason, learn from experience, be told what to do, explain what 
they are doing, reflect on their experience and respond to surprise 
(http ://www.ai.sri. com/proj ect/CALO). 

CARTE (Center for Advanced Research in Technology for Education). US research project 
based at the University of Southern California’s Information Sciences Institute (ISI). 
Developing tools to support individualized language learning 
(http://www.isi.edu/isd/carte/proj_tactlang/). 

Cathexis. A computational model for the generation of emotions and their influence in the 
behavior of autonomous agents (see Velasquez and Maes 1997). 

COMPANIONS: European Commission project developing conversational artificial 
Companions (including two demonstrators: a Senior Companion for elderly people and a 
Health and Fitness Companion). The COMPANIONS demonstrators will be agents or 
‘presences’ that stay with the user for long periods of time, developing a relationship and 
‘knowing’ its owners preferences (http://www.Companions-project.org). 
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CoSY (Cognitive Systems for Cognitive Assistants). European Commission project involving 
seven centres in constructing physically instantiated systems that can perceive, understand 
and interact with their environment. They will also be able to evolve to achieve human-like 
performance in activities requiring context- and task-specific knowledge 
(http://www.cs.bham.ac.uk/research/projects/cosy/index.php). 

Emotionally Intelligent Interface. Research led by Peter Robinson at Cambridge University. 
Provides a taxonomy of facial expressions and the emotions they represent, which has been 
used as the basis of a ‘Mind Reading’ DVD: an interactive computer-based guide to reading 
emotions from the face and voice (http://www.cl.cam.ac.uk/research/rainbow/emotions/). 

HUMAINE (Human-Machine Interaction Network on Emotion). European Commission 
project developing ‘emotion-oriented systems’ that can register, model and/or influence 
human emotional and emotion-related states and processes ( http ://emotion-research.nef) . 

INDIGO (Interaction with personality and dialogue-enabled robots). A European 
Commission project developing technology that could enable robots to perceive natural 
human behavior and to act in ways that are familiar to humans 
(http://www.ics.forth.gr/indigo/) 

Oz. Completed project at Carnegie Melon University that developed technology and art 
approaches to help create high quality interactive drama, based in part on AI technologies 
(http://www.cs.cmu.edu/afs/cs.cmu.edu/project/oz/web/oz.html). 
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RobotCub. A European Commission project which aims to build an open-source humanoid 
robot platform, such as KASPAR, for cognitive development research 
(http ://www.robotcub. org). 

Robot ethics. Roboethics.org is a group coordinating work on the application of ethics to the 
development and use of robots (http://www.roboethics.org). Aaron Sloman has led research 
in this field at the School of Computer Science, Birmingham University (e.g. see Sloman 
1978 and http://www.cs.bham.ac.uk/research/projects/cogaff/crp/epilogue.html). 

Sheffield University Department of Computer Science. Includes a range of work related to 
artificial Companions, including cognitive systems, natural language processing and speech 
technology (http://www.shef.ac.uk/dcs/research) 

UTOPIA. A consortium of Dundee, Abertay, Glasgow and Napier Universities funded by the 
Scottish Higher Education Funding Council to research the relationship between older people 
and technology (http://www.computing.dundee.ac.uk/projects/UTOPlA/). Notes 
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Notes 


1 I have studied people’s attitudes toward computational culture for over a quarter century, in particular the way 
computational objects become carriers for ideas about psychology, mind, the nature of relationship. On how this 
process plays out for personal computers and the Internet, see for example. Sherry Turkle, The Second Self: 
Computers and the Human Spirit (Cambridge, Mass.: MIT Press, 2005[1985] and Sherry Turkle, Life on the 
Screen: Identity and the Age of the Internet (New York: Simon and Schuster, 1995). For the past fifteen years, 
the focus of my work has been the nascent robotics culture. See, for example, Turkle, “First Encounters with 
Kismet and Cog: Children Respond to Relational Artifacts." With Cynthia Breazeal, Olivia Daste, and Brian 
Scassellati. In Digital Media: Transformations in Human Communication, Paul Messaris and Lee Humphreys 
(eds.). New York: Peter Lang Publishing, 2006; “Relational Artifacts with Children and Elders: The 
Complexities of CyberCompanionship” (with Will Taggart, Cory D. Kidd, and Olivia Daste), Connection 
Science, Vol. 18 No. 4, December 2006; 

“Authenticity in the Age of Digital Companions," Interaction Studies, Vol. 8 No. 3, 2007 (special issue on 
Psychological Benchmarks of Human-Robot Interaction). 

2 Turkle, The Second Self. 

3 Ibid. 

4 Turkle, Life on the Screen. There I discuss the case of computer psychotherapy to illustrate a turning away 
from the romantic reaction as a response to programs as appropriate dialogue partners in the realm of the 
personal. 

5 Ibid. 

6 Suvendi Kakushi, “Robot Lovin’” Asia Week Magazine Online, November 9, 2001. 
http://www.asiaweek.eom/asiaweek/magazine/life/0.8782.182326.00html . Accessed on 5/9/05. 

7 Interaction Studies, Ibid. 

s David Levy, Love and Sex With Robots: The Evolution of Human-Robot Relationships (New York: 
HarperCollins, 2007). 

9 Sigmund Freud, “The Uncanny,” In The Standard Edition of the Complete Works of Sigmund Freud, edited by 
James Strachey, et al. London: The Hogarth Press and The Institute of Psychoanalysis, 1953-74. 

10 P. H. Ornstein, Ed., The search for the self: Selected writings of Heinz Kohut: 1950-1978, (Vol. 2). New 
York: International Universities Press, Inc., 1978) 

11 On this, see Sherry Turkle, “Whither Psychoanalysis in Computer Culture,” Psychoanalytic Psychology’, Vol. 
21,2004. 

12 See for example Wilks (2005), Wilks (2006), the European projects on AC developed by the Adaptive 
Systems Research Group of the University of Hertfordshire on Human-Robot Interaction, 
http://adapsvs.feis.herts.ac.uk/ . 

13 This paper is a revised version of Floridi (2008). 

14 Source: American Pet Products Manufacturers Association, http://www.appma.org/press_industrytrends.asp 

15 Source: Entertainment Software Association, http://www.theesa.com/facts/top_10_facts.php 

16 Source: Neopets, http://www.neopets.com/petcentral.phtml 

17 Source: Motability, http://www.motability.co.uk/ 

ls Source: “The Expanding Digital Universe: A Forecast of Worldwide Information Growth Through 2010” 
white paper sponsored by EMC—IDC http://www.emc.com/about/destination/digital_universe/ 

19 (NB No relation: he spells his name wrongly. 

20 Of course, it is highly misleading to say that we ‘apply standards’ to humans to determine that they have 
human intelligence. It is not usually true to say that our interactions with other humans commence only when 
they have passed some test or exam. It only occurs to us to even think that we might have such standards when 
we find ourselves in highly unusual contexts when the existence of intelligence is in question - such as when we 
find ourselves judging for the Loebner Prize. Note that the standards to which I refer here are standards to 
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determine whether the object has a ‘human’ intelligence at all; there are several standards by which we judge the 
level of an already accepted human intelligence. 

21 ‘Intelligent’ can appear in scare quotes if the reader is sceptical about machine intelligence or keen to 
distinguish it from that of a human. In this case, the machine intelligence would consist, at a minimum, in 
abilities to (a) parse and make sense of human conversation, (b) adapt user models in the face of input 
information, and (c) develop courses of action that serve the original person’s interests. Depending on the level 
of trust involved, the Companion might also need to be able to manipulate the original’s passwords, or even 
bank accounts. The main thing is that the original cannot instruct the Companion at any level of detail precisely 
because ex hypothesi the user is unable or unwilling to attend to the detail of his or her online interactions. If the 
original could do that, he or she would not need the Companion in the first place. In AI terminology, the 
interactions between original and Companion take place at the knowledge level. 

22 http://www.Companions-proiect.org/ . 

23 Disconcertingly, for the British viewer of a certain age, without Doddy’s distinctive Liverpool accent.. 

24 http://www.ai.mit.edu/proiects/humanoid-robotics-group/kismet/kismet.html . 

25 Likely but not necessarily. The technology required to go beyond the furry handbag to produce a Companion 
physically linked to the original is perhaps more like Haraway’s conception of a cyborg. 

26 There are many uncertainties here, but this general statement sums up the rough consensus. 

27 Or, alternatively, ‘modernized’. 

28 One novel ( The Deer Park) and a couple of short stories remain of this scheme. One reason for the failure of 
Mailer’s Balzacian project was that he gradually morphed into this idealised observer figure himself, complete 
with fake Irishisms, with the projected novels replaced by long pieces of journalism in the first person - an 
indication of the closeness of the literary ‘Companions’ and the original in this case. 

29 In Blade Runner, the protagonist Deckard falls in love with a robot called Rachael. There has been 
debate among film buffs as to whether Deckard himself is meant to be a robot, in which case this would be a 
robot-robot relationship rather than a human-robot relationship. The director, Ridley Scott, has gone on record 
stating that Deckard is a robot, but neither version of the film (the original version or the director's cut) is 
explicit on this point. 

30 The First International Conference on Human-Robot Personal Relationships was held at Maastricht 
University on 12-13 June 2008. This paper was first presented at the conference. 

31 I cannot find the original source of this saying. The earliest variant of it I can find is in the works of 
the Earl of Chesterfield: “Marriage is the cure of love, and friendship the cure of marriage.” (Chesterfield, 
2005:361). 

32 E.g. Work at Newcastle University’s Institute for Ageing and Health 

33 Or as economists would put it: If you have preferences, rather than simply unrationalisable choice behaviors 

34 See e.g. K. N. Kirby and B. Guastello (2001) Making choices in anticipation of future similar choices can 
increase self- control. Journal of Experimental Psychology: Applied, 7, pp. 154-164. 

35 See e.g. S. McClure, K. Ericson, D. Laibson, G. Loewenstein, and J. Cohen (2007) Time discounting for 
primary rewards. Journal of Neuroscience, 27, pp.5796-5804 

36 See e.g. G. Ainslie (2005) Pr'ecis of Breakdown of Will, Behavioral and Brain Sciences 28, pp.635-673; G. 
Ainslie (1991). Derivation of “rational” economic behavior from hyperbolic discount curves, American 
Economic Review 81, pp.334-340 

37 See e.g. C. Harris and D. Laibson (2001) Dynamic choices of hyperbolic consumers Econometrica, 69(4), 
pp.935-957 

38 R. Nozick (1969) ‘Newcomb’s Problem and Two Principles of Choice’, in Essays in Honor of C. G. Hempel, 
N. Rescher et al. (ed.) Reidl, Dordrecht 

39 J. Bentham (1995) The Panopticon Writings. M. Bozovic (ed.) Verso, London 

40 Expect to see people shopping around for Companions with ideologically appealing frameworks. 

41 See e.g. P. Streich and J. S. Levy (2007) Time horizons, discounting, and inter-temporal choice, Journal of 
Conflict Resolution, 51 pp. 199-226 

42 http://thesims.ea.com/) 

43 http://secondlife.com/ 

44 http://www.arclight.net/~pdb/nonfiction/uncanny-valley.html 
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4 ' The Socio-Digital Systems group was initiated in 2004 to examine the intersection between social life 
and technology. Further details can be found here: http://research.microsoft.com/sds/. 

1 These debates have their origins in the early ideas of intelligent machinery set out by Alan Turing 

(1950) and counter-arguments posed by Ludwig Wittgenstein (Shanker 1998). 

4 That is, part of those ‘determinative adjectives’ that philosopher Gilbert Ryle associates with the 

“family of concepts ordinarily surnamed ‘intelligence’.” (1949: 26) 

4 The inspiration for this renewed perspective came about from readings of Wittgenstein (1973) and Ryle 

(1949) and, more generally, the philosophy or ordinary language, cf. Harris (1981) and Hanfling (2000). 

4 Nigel Thrift (2003) writes convincingly on the influence toy design has had and is likely to have on the 

developments of machine intelligence. He also speculates on how these same developments will shape our ideas 
of being human. Thrift also observes that significant transformations in software—based on biological models— 
are making devices “increasingly ‘lively’” (2004: 461). He suggests, although unnoticed, these changes are set 
to radically alter our relations with machines. 

The terms Al-complete and Al-hard are used to refer to the foundational problems associated with 
building computers to be as intelligent as humans. In doing so, they reflect one of the underlying motivations in 
AI and, in particular, GOFAI. 

We reference these distinct strands of work to highlight the significant efforts to reconsider some of 
established ideas of mind, body and agency. However, we are mindful of the distinctions between them. For 
example, by relocating ‘thinking’ beyond the mind, Clark and Chalmers (1998) are arguably just distributing the 
mental act (Dreyfus 2007), whereas others claim to be radically overhauling it’s ontology (e.g., Haraway 1997). 

Science and technology studies (STS) is beginning to contribute in a number of ways to the 
understanding of design and design practice, cf. Latour & Weibel, (2005), Kimbell (2008). Contributions to the 
arts and other design settings, is seen as one way for STS to actively participate in and impact on wider 
disciplinary and public discourses. They also allow STS to involve itself in the materialism that it has made its 
business to study. Our own efforts in speculative design have been heavily influenced by the designers Anthony 
Dunne and Fiona Raby, and their ideas on critical design or design for debate (Dunne 2006; Dunne & Raby 
2002 ). 
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55 

56 

57 


Summer Workshop, eNTERFACE project, France 2008; http://enterface08.limsi.fr/ 

FP7 STREP SEMAINE project IST-211486, http://www.semaine-project.eu 
FP6 IP CALLAS project IST-034800, http://www.callas-newmedia.eu 
http://wiki.mindmakers. 0 rg/pr 0 jects:SAIBA:main 

MARY text-to-speech system developed by March Schroder, DFKI. http://marv.dfki.de/ 


Work done in collaboration with Marc Price, BBC. 

60 Jerome Urbain and Thierry Dutoit, Faculte Polytechnique de Mons, Belgium 

Stelios Asteriadis and Amaryllis Raouzaiou, Image, Video and Multimedia Systems Laboratory, 
National Technical University of Athens, Greece 

62 http://wikitravel.org/ 

63 http://www.virtualtourist.com/ 

64 E.g. the BigDog and LittleDog robots of Boston Dynamics. 

65 In Sloman (1978) an argument was presented that it is possible for machines to have their own goals, 
preferences and values if, like humans, instead of being given only fixed high level goals, they start off with 
mechanisms for absorbing, generating, and modifying their goals, preferences and values during development. 
When such machines have their own goals, treating them as slaves, as proposed by one of the chapter authors in 
this volume, would be highly unethical, like racism. 

66 Some of the issues involved in this were discussed in Will Lowe’s contribution to this volume. . 

67 E.g. A is closer to B than C is to D. 

68 Available online at http://www.cs.bham.ac.Uk/research/proiects/cosv/papersA#dp0702 

69 Available here: http://www.cs.bham.ac.uk/research/proiects/cogaff/crp/epilogue.html 

70 See http://www.cs.bham.ac.uk/research/proiects/cogaff/misc/asimov-three-laws.html 
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71 A 'meme' is defined as a unit of cultural transmission. 

72 But note again the asymmetry: human culture is understandable to our AC robots because we have designed 
them that way. 

7j The Learning Companion: an embodied conversational agent for learning is a collaborative research project 
between the Oxford Internet Institute and the Department of Education, University of Oxford. It is supported by 
the OUP Fell Fund. Please see http://www.oii.ox,ac.uk/research/proiect.cfm?id=52 for more details. 

1 1. A robot may not injure a human being or, through inaction, allow a human being to come to hann. 2. A 
robot must obey orders given to it by human beings, except where such orders would conflict with the First 
Faw. 3. A robot must protect its own existence as long as such protection does not conflict with the First or 
Second Faw. 

2 An interesting situation that arises from following Asimov’s premises is that the Companions must not follow 
human commands without first reasoning whether such actions would cause harm to humans (including, 
possibly, the one to whom they are Companion). The Companions must be made to follow the principle of 
adjustable autonomy, following the human’s lead in some cases and overriding it in others. 

3 Curiously, most of the authors of this volume have deemed embodiment (or at least realistic embodiment) an 
expendable property for a Companion, even though it is common knowledge that small furry creatures and their 
renditions in plush make many people happy. This, I think, is a prudent result of prioritizing the work in the 
area: functionality first, appearances later. The rationale for deeming embodiment less important is the correct 
belief in our facility for accepting genre conventions, as in the theater or in the plastic arts (if we don’t count 
deliberate violations of the rules of the game - we would want the users of Companions to believe that it is a 
pipe in Magritte’s painting, not just the picture of a pipe). Of course, acclimation to genre conventions relates to 
issues beyond embodiment. By accepting a set of conventions users create a set of expectations for the 
Companions and adapt to them. 

4 This will be necessary even if one accepts Lowe’s (this volume) opinion that users will prefer Companions that 
are entirely like themselves. 

5 Minsky believes that “a clear definition can make things worse, until we are sure that our ideas are right” 
(op.cit.: 95). 

6 The choice of using mature methods versus working on perfecting immature ones is certainly not unique to 
work on Companions. For example, it is similar to the 1960s choice between 95% and 100% approaches to 
machine translation system building. 

74 This latter ability, incidentally, will bring about collateral benefits - one of the ways many people needing 
Companions could be made happy is to for them to be made to feel useful. This goal may be at least partially 
satisfied if they can teach their Companion new facts, new ways of doing things and, last but not least, new or 
better understanding of their own preferences and traits. 

73 This discussion of the agent’s functionalities is rather coarse-grained. Some more detail can be found in 
(Nirenburg et al. 2008a,b; McShane and Nirenburg 2009). 

76 http://www.reuters.com/news/globalcoverage/agingiapan 

77 http://www.google.com/hostednews/afp/article/AFeqM5iuSqhZrvHpsVuY6mf93nr92glqdA 

78 See Appendix 1 and the Reference section for information on related research, products and publications 
providing further background to the issues raised in this chapter. 

79 See, for example, Cassell et al (2000a) and Pelachaud (2005; 2007). 

80 See, for instance, the work of MIT’s Affective Computing Research Group 
( http://affect.media.mit.edu/index.php http://affect.media.mit.edu/people.php?id=hoda) . 

81 See, for example, Sloman (2006), the HUMAINE research network and the Emotionally Intelligent Interface 
research led by Peter Robinson, Cambridge University based on the theory of mind of Professor Simon Baron- 
Cohcn, Director of the Autism Research Centre at Cambridge 
( http://www.cl.cam.ac.uk/research/rainbow/emotions/) . 

82 See, for example, MIT’s Humanoid Robotics Group ( http://www.ai.mit.edu/proiects/humanoid-robotics- 
group/index.html ). 

83 See http://www.Companions-proiect.org for details of the COMPANIONS project. An example of an AC 
being developed for it is at: http://www.asanangel.fr/morgan/ 

84 For example, see Berners-Fee et al. (2001) for a discussion on the emergence of the concept of a Semantic 
Web. 

85 Bryson et al (2002) propose that the semantic web (particularly Web services) could be thought of neither as 
passive content nor other agents to be negotiated with, but rather as extensions to the mind of an agent’s user. 
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Users determine their agents’ motivational structure and communicate their needs to it, then the Web provides 
the agent with capacities to better meet those needs. 

S6 See http://www2.napier.ac.uk/Companions/Movie%203.html for an example of a COMPANION’S approach 
to finding out the requirements of senior citizens. 

87 See http://www.youtube.com/watch?v=KOSiigSEYhU for a video of an early prototype. 

88 See also, for example, http://www.memoriesforlife.org regarding the UK EPSRC research project Memories 
for Life (M4L), which is seeking to help define and solve the problems caused by people storing increasingly 
large quantities of information about themselves using digital versions of life’s memories (in the form of 
photographs, documents, video, etc). 

89 For more on NLP and related areas see, for example, research at COMPANIONS partners at the University of 
Sheffield NLP Group l http://nlp.shef.ac.uk ) and University of Oxford’s Department of Linguistics and 
Phonetics ( http://www.clg.ox.ac.uk ). The speech technology research of Roger Moore at the University of 
Sheffield, part of the COMPANION’S team, was also highlighted at the forum (see Zyga 2007 and 
http://www.dcs.shef.ac.uk/~roger/) . 

90 See, for instance, Birmingham University’s 3D vision research 
( http://www.cs.bham.ac.Uk/research/proiects/cogaff/talks/#compmod07 ) 

91 See, for example, Rehm and Andre (2005). 

92 For further background on AI, see also: http://www.aaai.org/AITopics/html/welcome.html 

93 For more background on the ideas developed by Turkic, see her seminal book The Second Self (Turkic 2005). 

94 See http://news.bbc.co.Uk/l/hi/programmes/this world/golden vears/4373857.stm 

93 See also Rice et al (2007) and http://www2.napier.ac.uk/Companions/Movie%203.html regarding such use of 
theatre in which Newell has been involved, as well as, for instance, the technology and art approaches 
developed for Carnegie-Mcllon University’s Oz Project 
( http://www.cs.cmu.edu/afs/cs.cmu.edu/proiect/oz/web/oz.html) . 

96 For example, users engaging in conversation with the Virtual Woman 3D animated image (see Appendix 1) 
can choose from a range of characteristics for its presentation. 

97 The time scale on enabling robots to operate with physical, as well as mental, human-like capabilities is 
indicated by the international RoboCup challenge, which aims to develop a team of fully autonomous humanoid 
robots that can win against the human world champion team in soccer—by the year 2050 (see 
http://www.robocup.org ). 

98 See for example the range of topics being investigated at the 011 ( http://www.oii.ox.ac.uk) and Oxford 
University e-Horizons Institute ( http://www.e-horizons.ox.ac.uk) . 

99 See: http://www.billzarchv.com/clips/clips apple nav.htm 

100 See also Weizenbaum (1984) for a discussion on the relatively primitive Eliza software program that engaged 
in what some users thought were believable psychiatric counselling. 

101 On robot ethics and rights, see also Robocthics.org ( http://www.roboethics.org ) and research at Birmingham 
University (e.g. http://www.cs.bham.ac.uk/research/proiects/cogafl/crp/epilogue.html and Sloman 1978). 

102 For instance, see Ecobot (Appendix 1). 
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